Package tracking method based on X-ray image and visible light image

Abstract

The present invention provides a package tracking method based on X-ray images and visible light images, the method comprising: when a target contraband is detected based on an X-ray image, intercepting a first package image including the target contraband from a first visible light image; the X-ray image and the first visible light image are shot at the same time; matching the first package image with at least one second package image, the second package image being an image of a package within each package tracking frame in at least one second visible light image shot after the first visible light image; and tracking the package corresponding to the first package image based on the matching result. The present invention can improve security inspection efficiency and security inspection reliability.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a package tracking method based on X-ray images and visible light images. Background Art

[0002] Preventing passengers from bringing prohibited items into public places can eliminate most safety incidents caused by accidents at the root.

[0003] At present, the detection of metal objects in passengers' packages and the identification of various types of knives in them through X-ray machines and handheld security inspection equipment have been widely used. For example, target detection algorithms such as the Yolo series algorithm can be used to detect and identify prohibited items in X-ray images. When the target detection algorithm is used to detect X-ray images and find prohibited items, the shape of the package in the X-ray image is matched with the actual package, and the package carrying prohibited items is manually sorted out from many packages.

[0004] However, there are significant differences between X-ray images and visible light images. It is difficult to manually match items in X-ray images with their corresponding actual items, especially during peak passenger flow periods. When multiple packages are mixed and pass through the security inspection machine during the security inspection process, it is even more difficult to use the obtained X-ray images to achieve the above-mentioned package matching, which not only reduces the efficiency of security inspection, but also causes missed inspections, making the reliability of security inspection low. Summary of the invention

[0005] The present invention provides a package tracking method based on X-ray images and visible light images, which is used to solve the defects of low security inspection efficiency and low reliability in the prior art, and achieves the purpose of improving security inspection efficiency and security inspection reliability.

[0006] The present invention provides a package tracking method based on X-ray images and visible light images, comprising:

[0007] In the case where a target contraband is detected based on the X-ray image, a first package image including the target contraband is intercepted from the first visible light image; the X-ray image and the first visible light image are taken at the same time;

[0008] matching the first package image with at least one second package image, where the second package image is an image of a package within each package tracking frame in at least one second visible light image taken after the first visible light image;

[0009] Based on the matching result, the package corresponding to the first package image is tracked.

[0010] The present invention also provides a package tracking device based on X-ray images and visible light images, comprising:

[0011] An interception module, configured to intercept a first package image including the target contraband from a first visible light image when the target contraband is detected based on the X-ray image; the X-ray image and the first visible light image are taken at the same time;

[0012] a matching module, configured to match the first package image with at least one second package image, where the second package image is an image of a package within each package tracking frame in at least one second visible light image taken after the first visible light image;

[0013] A tracking module is used to track the package corresponding to the first package image based on the matching result.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a package tracking method based on X-ray images and visible light images as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the package tracking method based on X-ray images and visible light images as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for tracking packages based on X-ray images and visible light images.

[0017] The package tracking method based on X-ray images and visible light images provided by the present invention intercepts a first package image including the target contraband from a first visible light image taken at the same time as the X-ray image when the target contraband is detected based on the X-ray image, and matches the first package image with the second package image corresponding to each package in at least one second visible light image taken after the first visible light image, thereby tracking the package containing the target contraband according to the matching result. Since when a package is detected to contain the target contraband by the X-ray image, the visible light image containing more feature information such as the package appearance can be used to track the package, thereby achieving rapid matching and tracking of the package, thereby improving the efficiency of security inspection. In addition, it can also avoid the phenomenon of missing packages due to the inability to distinguish which package contains the target contraband only through the X-ray image, thereby improving the reliability of security inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of a process flow of a package tracking method based on X-ray images and visible light images provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of matching a first package image and a second package image provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of package tracking provided by an embodiment of the present invention;

[0022] Figure 4 An X-ray image provided by an embodiment of the present invention;

[0023] Figure 5 A visible light image provided by an embodiment of the present invention;

[0024] Figure 6 A schematic diagram of a first package image provided by an embodiment of the present invention;

[0025] Figure 7 One of the schematic diagrams of package matching inside and outside the channel provided by an embodiment of the present invention;

[0026] Figure 8 The second schematic diagram of the package matching inside and outside the channel provided by the embodiment of the present invention;

[0027] Fig. 9 It is a schematic diagram of the structure of the conv-5 layer of resnet50 in the prior art;

[0028] Fig.10 A schematic diagram of the structure of the conv-5 layer of the improved resnet50 provided in an embodiment of the present invention;

[0029] Fig.11 A schematic diagram of the structure of a package tracking device based on X-ray images and visible light images provided by an embodiment of the present invention;

[0030] Fig.12 An example physical structure diagram of an electronic device is provided. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In order to prevent passengers from carrying prohibited items into public places with high security requirements such as stations and airports, X-ray security inspection technology is currently widely used to detect the packages carried by passengers. In addition, with the development of artificial intelligence, deep learning methods have also been widely used in X-ray security inspection technology. Through deep learning, X-ray images can be automatically recognized and classified, greatly improving the efficiency and accuracy of security inspection. In X-ray security inspection technology, deep learning methods usually use convolutional neural networks (CNN) for image recognition. In addition to convolutional neural networks, some other deep learning methods are also applied to X-ray security inspection technology, such as generative adversarial networks (GAN). These methods can improve the accuracy and reliability of security inspection by generating realistic X-ray images. In the latest technology, target detection models such as yolov5 are widely used for target detection, which can not only identify contraband, but also specifically classify the type, quantity and specific location of contraband.

[0033] At present, when prohibited items are found after detecting X-ray images using target detection algorithms, the shape of the package in the X-ray image is matched with the actual package, and the package carrying the prohibited items is manually sorted out from the many packages. However, due to the large difference between X-ray images and visible light images, it is difficult to manually match the X-ray image with the corresponding package, especially when there are many and dense packages during peak hours. It is even more difficult to achieve the above-mentioned package matching using the obtained X-ray image, which not only reduces the efficiency of security inspection, but also causes missed inspections, making the reliability of security inspection low.

[0034] In consideration of the above problems, an embodiment of the present invention proposes a package tracking method based on X-ray images and visible light images. In the method, when a target prohibited item is detected based on an X-ray image, a first package image including the target prohibited item can be intercepted from a first visible light image taken at the same time as the X-ray image, and the first package image can be matched with a second package image of each package in at least one second visible light image taken after the first visible light image, so that the package containing the target prohibited item can be tracked according to the matching result. Since when a package is detected to contain the target prohibited item through an X-ray image, the package can be tracked using a visible light image containing more feature information such as the package appearance, so that the package can be quickly matched and tracked, thereby improving the efficiency of security inspection. In addition, the phenomenon of missing packages due to the inability to distinguish which package contains the target prohibited item only through the X-ray image can be avoided, thereby improving the reliability of security inspection.

[0035] It should be noted that the parcels mentioned in the embodiments of the present invention may include express deliveries, mail items, and various luggage items in security inspection scenarios, and prohibited items may be understood as prohibited or restricted items in security inspection scenarios.

[0036] Combine the following Figures 1 to 10 The package tracking method based on X-ray images and visible light images provided in the embodiment of the present invention is described. The package tracking method based on X-ray images and visible light images provided in the embodiment of the present invention can be applied to any scenario where it is necessary to identify whether the package carries prohibited items, such as subways and airports. The execution subject of this method can be an electronic device such as an X-ray security inspection machine, a prohibited item identification machine, a computer or a server, or a specially designed intelligent device, or a package tracking device based on X-ray images and visible light images set in the electronic device or the intelligent device. The package tracking device based on X-ray images and visible light images can be implemented by software, hardware, or a combination of the two.

[0037] Figure 1 A schematic diagram of a package tracking method based on X-ray images and visible light images provided in an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0038] Step 101: When a target contraband is detected based on an X-ray image, a first package image including the target contraband is intercepted from a first visible light image; the X-ray image and the first visible light image are shot at the same time.

[0039] In this step, after the X-ray image is captured by the X-ray machine, it can be determined based on the X-ray image whether the currently captured package contains the target prohibited items. The X-ray machine can be a dual-source or multi-source X-ray machine, and the X-ray image is obtained after the dual-source or multi-source X-ray machine scans the package carried by the passenger.

[0040] Exemplarily, the X-ray image can be input into the Yolov8 target detection model for contraband detection, wherein Yolov8 is a target detection algorithm based on a deep convolutional neural network, which is improved from the Yolov5 model.

[0041] Yolov8 is mainly composed of three parts: Backbone, Neck and Head. Among them, the Backbone part is responsible for feature extraction, including Focus, CBS, BottleneckCSP and Spatial Pyramid Pooling (SPP) and other components. The Focus structure completes downsampling, which can reduce the amount of calculation and speed up the network while suppressing overfitting. After the original 640×640×3 image is input into the Focus structure, it is converted into a 320×320×12 feature map through slicing operation, and then concatenated and convolved, and finally becomes a 320×320×64 feature map. The CSP structure divides the original input into two branches, and performs convolution operations on each branch to halve the number of channels. One branch performs Bottleneck*N operation, and then concats the two branches together, so that the input and output of BottlenneckCSP are the same size, allowing the model to learn more features. The Neck part of Yolov8 is mainly composed of CBS (SiLU), Upsample, Concat and Cross Stage Partial Connections (CSP) without shortcuts, and follows the Feature Pyramid Network (FPN) + Path Aggregation Network (PAN) structure. The main function of the Neck part is to make better use of the features extracted by Backbone and play the role of "feature fusion". Through components such as CBS, Upsample and Concat, the Neck part can fuse feature maps at different levels, improve the richness and expression ability of features, and provide more accurate information for subsequent target detection. The Head part is responsible for the final output of the detection results, and its main structure includes Decoupled-Head and Anchor-Free Loss.

[0042] After obtaining the X-ray image of the package entering the X-ray machine channel, the above-mentioned target detection model is used to detect whether the X-ray image carries contraband. If there is no contraband in the package, no processing is performed on the package. If a package is detected to contain target contraband based on the X-ray image, the first visible light image taken at the same time as the X-ray image is obtained, and based on the position of the target contraband in the X-ray image, the position of the package containing the target contraband in the first visible light image is determined, and the first package image containing the target contraband is intercepted from the first visible light image. Among them, the first package image includes the package containing the target contraband.

[0043] Step 102: Matching the first package image with at least one second package image, where the second package image is a package image within each package tracking frame in at least one second visible light image taken after the first visible light image.

[0044] In this step, Figure 2 A schematic diagram of matching a first package image with a second package image provided by an embodiment of the present invention, such as Figure 2 As shown, inside the X-ray machine, a security inspection video can be collected by a camera device, and the security inspection video includes multiple frames of ordered visible light images sorted according to the collection time. Based on the collected security inspection video, at least one second visible light image taken after the first visible light image can be obtained, and the target tracking algorithm can be used to determine at least one package tracking frame in each second visible light image, wherein the package tracking frame can be represented as (x1, y1, x2, y2), (x1, y1) represents the coordinates of the upper left corner vertex of the package tracking frame, and (x2, y2) represents the coordinates of the lower right corner vertex of the package tracking frame. It should be understood that the camera device used for contraband detection and the camera device used for tracking packages in the X-ray machine channel are the same imaging device, and the package target tracking algorithm will also be loaded on the camera device.

[0045] Figure 3 A schematic diagram of package tracking provided by an embodiment of the present invention, such as Figure 3 As shown in the figure, similar to the target detection algorithm, the package tracking algorithm will also obtain the package. The package tracking algorithm only performs package detection on the second visible light image to obtain the package detection frame, which is also expressed as a package detection frame. Unlike the package detection algorithm, the package tracking algorithm can also obtain the package ID of the package. In the security inspection video, the same package always has the same package ID, such as Figure 3 As shown, if the package ID is 0.37 or 0.33.

[0046] Exemplarily, in an embodiment of the present invention, Yolov8 tracker can be used for package tracking, wherein Yolov8 tracker is a target tracking algorithm based on deep learning, which inherits the idea of ​​target detection algorithm of Yolov series and realizes detection and tracking of targets in security inspection videos. Yolov8 tracker uses a multi-target tracking algorithm called BoT-SORT, which combines the advantages of SORT algorithm and BoF (Bag of Features) algorithm and can realize fast and accurate target tracking.

[0047] Continue to refer to Figure 2 As shown, for each second visible light image, after determining at least one package tracking frame in the second visible light image, the package image in each package tracking frame is intercepted to obtain at least one second package image, so as to match the first package image containing the target contraband with each second package image obtained after tracking.

[0048] Step 103: Based on the matching result, the package corresponding to the first package image is tracked.

[0049] In this step, based on the matching results between the first package image containing the target prohibited items and the second package images obtained through tracking, the package corresponding to the first package image can be tracked. For example, when the first package image containing the target prohibited items successfully matches a second package image, it can be quickly determined that the second package image and the first package image represent the same package, so that it can be determined that the package included in the second package image contains the target prohibited items. In this way, the security inspector can quickly locate the package represented by the second package image, avoiding the phenomenon of missed inspection.

[0050] The package tracking method based on X-ray images and visible light images provided in the embodiment of the present invention intercepts a first package image including the target contraband from a first visible light image taken at the same time as the X-ray image when the target contraband is detected based on the X-ray image, and matches the first package image with the second package image corresponding to each package in at least one second visible light image taken after the first visible light image, thereby tracking the package containing the target contraband according to the matching result. Since when a package is detected to contain the target contraband by the X-ray image, the visible light image containing more feature information such as the package appearance can be used to track the package, thereby achieving fast matching and tracking of the package, thereby improving the efficiency of security inspection. In addition, it can also avoid the phenomenon of missing the package due to the inability to distinguish which package contains the target contraband only by the X-ray image, thereby improving the reliability of security inspection.

[0051] Exemplarily, based on the above embodiment, when intercepting the first package image including the target contraband from the first visible light image, it can be performed in the following manner:

[0052] Determine the first center point coordinates of the target detection frame representing the target contraband in the X-ray image, and determine the second center point coordinates of each package detection frame in the first visible light image, determine the target distance between the first center point coordinates and the second center point coordinates for each second center point coordinate, and after intercepting the package detection frame to which the second center point coordinate corresponding to the minimum target distance belongs, determine the image within the intercepted package detection frame as the first package image.

[0053] Specifically, Figure 4 The X-ray image provided by the embodiment of the present invention, Figure 5 The visible light image provided by the embodiment of the present invention is as follows: Figure 4 and Figure 5 As shown, based on the X-ray image, it can be detected whether the package contains the target prohibited items. If it is determined that the package contains the target prohibited items, it is impossible to determine which package in the visible light image the package containing the target prohibited items corresponds to. Among them, the package detection algorithm can detect all the packages in the security inspection video in real time. The package is presented in the form of a package detection frame for each package, that is, each package detection frame contains a package. In fact, in the X-ray machine, since the X-ray image will capture multiple packages, the visible light image will also detect multiple packages through the target detection algorithm. Therefore, the packages in the X-ray image and the packages in the visible light image will not correspond one to one.

[0054] In order to solve this problem, the embodiment of the present invention uses the real-time position of the package in the visible light image and the position of the target contraband in the X-ray image for matching. Specifically, the first center point coordinates of the target detection frame where the target contraband is located in the X-ray image can be determined, and the second center point coordinates of each package detection frame in the first visible light image can be determined. A set of corresponding X-ray images and the first visible light image are overlapped, and the target distance in the same plane between the first center point coordinates of the target detection frame where the target contraband is located in the X-ray image and the second center point coordinates of each package detection frame in the first visible light image is calculated, for example, the Euclidean distance between the two in the same plane can be determined. In the first visible light image, the package detection frame with the smallest Euclidean distance between the first center point coordinates of the target detection frame of the target contraband is selected, and the package corresponding to the package detection frame is determined as the package matching the package containing the target contraband, so that the image in the package detection frame is determined as the first package image. Figure 6 A schematic diagram of a first package image provided by an embodiment of the present invention, such as Figure 6As shown, the image in the package detection frame, that is, the first package image, can be captured from the first visible light image for storage. It should be understood that the first package image can be stored in a queue, and the timed first package image can be deleted from the queue regularly according to time.

[0055] In this embodiment, the target detection frame and each package detection frame are matched by the position of the target detection frame representing the target contraband in the X-ray image and the position of each package detection frame in the first visible light image. Since the position of the package hardly moves during the security check, when the package matching is performed based on the position of the above package detection frame, the package detection frame corresponding to the target detection frame of the target contraband can be accurately determined from the first visible light image, thereby improving the accuracy of the intercepted first package image.

[0056] Exemplarily, based on the above embodiments, when matching the first package image with at least one second package image, feature extraction may be performed on the first package image to obtain a first feature vector, and feature extraction may be performed on each second package image to obtain a second feature vector, a first grayscale distribution histogram corresponding to the first package image is obtained, and a second grayscale distribution histogram corresponding to each second package image is obtained, and the first package image is matched with the at least one second package image based on the first feature vector, each second feature vector, the first grayscale distribution histogram, and each second grayscale distribution histogram.

[0057] Among them, the ReID model can be used to match the feature vector. Specifically, in order to improve the image quality and the accuracy of the feature vector, the first package image and the second package image can be preprocessed, such as graying and size normalization, so as to facilitate the subsequent extraction of the feature vector. Further, deep learning technology can be used to extract features from the preprocessed first package image and the second package image through a feature extraction model. Among them, the feature extraction model is trained and extracted by a model such as a convolutional neural network. In an embodiment of the present invention, for example, a resnet50 model can be used as a feature extraction model. After the first package image is input into the resnet50 model, the first package image will be feature extracted, and the first feature vector corresponding to the first package image will be obtained, and the first feature vector will be flattened into one dimension to obtain a first feature vector with a length of 2048. Similarly, after each second package image is input into the resnet50 model, each second package image will be feature extracted, and the second feature vector corresponding to the second package image will be obtained, and the second feature vector will be flattened into one dimension to obtain a second feature vector with a length of 2048.

[0058] In addition, since the first package image and each second package image are taken by the same camera device, when matching the first package image and each second package image, combining the grayscale distribution histogram on the basis of the above-mentioned feature vector matching will improve the matching accuracy. Therefore, the first grayscale distribution histogram corresponding to the first package image and the second grayscale distribution histogram corresponding to each second package image can also be obtained and saved using the tools provided by opencv.

[0059] Furthermore, the first feature vector and each second feature vector may be matched, and the first grayscale distribution histogram and each second grayscale distribution histogram may be matched, and based on the matching result, an image that successfully matches the first package image may be determined from at least one second package image.

[0060] In this embodiment, since the feature vector and the grayscale distribution histogram can be combined to match the first package image and the at least one second package image, and since the package images are matched from multiple dimensions, the matching accuracy can be improved.

[0061] In a possible implementation, when matching the first package image with at least one second package image based on the first eigenvector, each second eigenvector, the first grayscale distribution histogram, and each second grayscale distribution histogram, a cosine distance between the first eigenvector and the second eigenvector may be determined for each second eigenvector, an approximate value between the first grayscale distribution histogram and the second grayscale distribution histogram corresponding to the second eigenvector may be determined, a target value may be determined based on the cosine distance and the approximate value, and a second package image corresponding to a minimum target value among all target values ​​may be determined as a package image matching the first package image.

[0062] Specifically, for each second eigenvector extracted from the second package image, the cosine distance cosine between the first eigenvector and the second eigenvector will be calculated, wherein the smaller the cosine distance, the more similar the two are. In addition, the opencv tool will be used to compare the first grayscale distribution histogram and the second grayscale distribution histogram corresponding to the second eigenvector to obtain the approximate value distance_col of the distribution histogram, wherein the smaller the approximate value, the more similar the two are.

[0063] Furthermore, according to the formula distance=cosine*k+distance_col*j, the cosine distance cosine and the approximate value distance_col can be multiplied by their respective weight coefficients k and j, and then the multiplied values ​​are added to obtain a weighted sum, and the weighted sum is determined as the target value distance. Among them, k and j are determined weight coefficients, and their specific values ​​can be set according to actual conditions, and k+j=1.

[0064] It should be understood that the smaller the target value distance is, the closer the first package image and the second package image are. Therefore, the second package image corresponding to the minimum target value among all target values ​​can be determined as the package image that matches the first package image containing the target contraband. In this way, the package containing the target contraband is found from the second visible light image and the package ID is obtained. In the specific implementation process, a list can be used to record the IDs of all packages containing the target contraband.

[0065] In this embodiment, a target value can be determined by weighted summing the cosine distance between the first eigenvector and each second eigenvector and the approximate value between the first grayscale distribution histogram and the second grayscale distribution histogram corresponding to the second eigenvector, and a second package image matching the first package image is determined based on the target value. Since the package images are matched in multiple dimensions, the matching accuracy can be improved.

[0066] Furthermore, the currently open-source package tracking of yolov8 can only be completed under a single camera, and in the X-ray machine security inspection equipment, there are lead curtains blocking the inside and outside of the X-ray machine channel, and it is impossible to track the package with only one camera. In actual situations, the package ID of the same package carrying contraband obtained by the target tracking algorithm in the X-ray machine channel may be different from the package ID of the same package carrying contraband obtained by the target tracking algorithm outside the channel, which will cause the package tracking to fail or the package to be lost.

[0067] In order to solve this problem, in an embodiment of the present invention, image matching and critical package matching methods can be used to pair the packages of the target tracking inside the channel with the packages of the target tracking outside the channel. Exemplarily, when the second visible light image is an image taken in an X-ray machine, the first identifier of each package tracking frame in the internal critical package status list can also be obtained, and the second identifier of each package tracking frame in the external critical package status list can be obtained; the internal critical package status list includes the tracking frame of the critical package that reaches the lead curtain of the X-ray machine determined based on the second visible light image; the external critical package status list includes the tracking frame of the critical package that leaves the lead curtain of the X-ray machine determined based on the third visible light image, and the third visible light image is an image taken outside the X-ray machine; for each first identifier, the third package image in the package tracking frame corresponding to the first identifier and the fourth package image in the package tracking frame corresponding to each second identifier are matched; it is determined that the package corresponding to the fourth package image that has been successfully matched and the package corresponding to the third package image are the same package.

[0068] Specifically, the second visible light image is an image captured by a camera device disposed inside the X-ray machine. The package tracking frame in each second visible light image can be determined by the target tracking algorithm, and based on the vertex coordinates of each package tracking frame, it can be determined whether the package tracking frame is a tracking frame of a critical package that reaches the lead curtain of the X-ray machine. If it is a tracking frame of a critical package that reaches the lead curtain of the X-ray machine, the first identifier of the package tracking frame can be added to the internal critical package status list. Among them, the package tracking frame added to the internal critical package status list may include a package tracking frame corresponding to a package carrying target contraband.

[0069] Exemplarily, when determining whether a package tracking frame is a tracking frame of a critical package reaching the lead curtain of an X-ray machine and adding the corresponding first identifier to the internal critical package status list, it can be determined whether the vertical coordinate of the lower right corner point coordinates of each package tracking frame in the second visible light image is equal to a first critical value, and the first critical value is used to characterize the critical position reaching the lead curtain of the X-ray machine; the package tracking frame whose vertical coordinate of the lower right corner point coordinates is equal to the first critical value is added to the internal critical package status list.

[0070] After determining each package tracking frame in the second visible light image through the target tracking algorithm, for each package tracking frame, assuming that the coordinates of the upper left corner point of the package tracking frame in the coordinate system are (x1, y1), and the coordinates of the lower right corner point are (x2, y2), when the ordinate y2 of the coordinates of the lower right corner point reaches the position of the set first critical value, it means that the lower edge of the package tracking frame is about to reach the lead curtain. At this time, the first identifier of the package tracking frame can be added to the internal critical package state list, and the package is marked as an internal critical package. Among them, the first critical value is used to characterize the critical position of the package reaching the lead curtain of the X-ray machine. It should be understood that since the camera device inside the X-ray machine cannot capture the lead curtain, when y2 reaches the first critical value, it may be at a certain actual physical distance from the lead curtain, while the camera device outside the X-ray machine can capture the lead curtain. Therefore, the position of the first critical value that the upper edge y1 can reach can be the actual position of the lead curtain.

[0071] In addition, when the package tracking algorithm fails to detect the package through the second visible light image, that is, the package completely leaves the X-ray machine channel, the first identifier of the package tracking frame corresponding to the package will be removed from the internal critical package status list.

[0072] In this embodiment, whether the package reaches the lead curtain position of the X-ray machine can be determined based only on the ordinate of the lower right corner point coordinate of each package tracking frame, thereby improving the efficiency of critical package determination.

[0073] In addition, due to the obstruction of the lead curtain of the X-ray machine, the camera device installed inside the X-ray machine will not be able to capture the package outside the lead curtain. Therefore, it is necessary to install a camera device outside the lead curtain of the X-ray machine. The camera device installed outside the X-ray machine can collect the third visible light image, and determine the package tracking frame in each third visible light image through the target tracking algorithm, and based on the vertex coordinates of each package tracking frame, it can be determined whether the package tracking frame is the tracking frame of the critical package leaving the lead curtain of the X-ray machine. If it is the tracking frame of the critical package leaving the lead curtain of the X-ray machine, the second identifier of the package tracking frame can be added to the external critical package status list. Among them, the package tracking frame added to the external critical package status list may include the package tracking frame corresponding to the package carrying the target contraband.

[0074] Exemplarily, when determining whether the package tracking frame is a tracking frame of a critical package leaving the lead curtain of an X-ray machine and adding the corresponding second identifier to the external critical package status list, it can be determined whether the vertical coordinate of the upper left corner point coordinates of each package tracking frame in the third visible light image is equal to the second critical value, the second critical value is used to characterize the critical position leaving the lead curtain of the X-ray machine, and the package tracking frame whose vertical coordinate of the upper left corner point coordinates is equal to the second critical value is added to the external critical package status list.

[0075] Specifically, after determining each package tracking frame in the third visible light image through the target tracking algorithm, for each package tracking frame, assume that the coordinates of the upper left corner point of the package tracking frame in the coordinate system are (x1', y1'), and the coordinates of the lower right corner point are (x2', y2'). When the ordinate y1' of the coordinates of the upper left corner point reaches the position of the second critical value, it means that the lower edge of the package tracking frame has just left the lead curtain. At this time, the package tracking frame can be added to the external critical package status list, and the package is marked as an external critical package. Among them, the second critical value is used to characterize the critical position of the package leaving the lead curtain of the X-ray machine.

[0076] In addition, when the ordinate y1' of the upper left corner point coordinate of the above-mentioned package tracking frame is greater than the second critical value, that is, when the upper edge of the package has left the lead curtain, it means that the package has completely left the X-ray machine channel. At this time, the second identifier of the package tracking frame corresponding to the package can be removed from the external critical package status list.

[0077] In this embodiment, whether the package leaves the lead curtain position of the X-ray machine can be determined based only on the ordinate of the upper left corner point coordinate of each package tracking frame, thereby improving the efficiency of critical package determination.

[0078] In summary, each package tracking frame in the internal critical package status list is a tracking frame corresponding to the package photographed inside the X-ray machine, and each package tracking frame in the external critical package status list is a tracking frame corresponding to the package photographed outside the X-ray machine.

[0079] When the number of first identifiers in the internal critical package status list and the number of second identifiers in the external critical package status list are both at least two, it is necessary to match the packages inside and outside the channel. Specifically, Figure 7 One of the schematic diagrams of the package matching inside and outside the channel provided by the embodiment of the present invention, such as Figure 7 As shown, when there are at least two internal critical packages and at least two external critical packages, for the first identifier of each package tracking frame in the internal critical package status list, the third package image in the package tracking frame corresponding to the first identifier and the fourth package image in the package tracking frame corresponding to each second identifier can be matched, for example, by extracting image features and calculating the cosine similarity or Euclidean distance between the features, the fourth package image that successfully matches the third package image can be determined. For the successfully matched fourth package image, the package corresponding to it and the package corresponding to the third package image are the same package, that is, they are regarded as two parts of the same package across the lead curtain, and the internal target tracking package id and the external target tracking package id are paired and recorded as [idi, idj], where the difference between i and j will gradually increase, idi represents the internal target tracking package id, idj represents the external target tracking package id, and the recorded identifier pair is stored in the pairing list for subsequent use in package matching and package sorting.

[0080] In this embodiment, the package tracking frame corresponding to the package inside the lead curtain of the X-ray machine and the package tracking frame corresponding to the package outside the lead curtain of the X-ray machine can be further matched, thereby avoiding the situation where the package tracking failure or package loss is caused by the package ID carrying contraband obtained by the target tracking algorithm for the same package in the X-ray machine channel being different from the package ID carrying contraband obtained by the target tracking algorithm outside the channel, thereby improving the accuracy of package tracking.

[0081] Exemplarily, when the number of first identifiers included in the internal critical package status list and the number of second identifiers included in the external critical package status list are both one, it is determined that the package corresponding to the fourth package image represented by the second identifier and the package corresponding to the third package image represented by the first identifier are the same package.

[0082] Specifically, Figure 8 The second schematic diagram of the package matching inside and outside the channel provided by the embodiment of the present invention is as follows: Figure 8As shown in FIG. 1 , when the ordinate y2 of the coordinate of the lower right corner point of the package tracking frame reaches the first critical value, the package corresponding to the package tracking frame will be marked as an internal critical package, and when the ordinate y1' of the coordinate of the upper left corner point of the package tracking frame reaches the second critical value, the package corresponding to the package tracking frame will be marked as an external critical package. If the number of first identifiers included in the internal critical package status list is 1, and the number of second identifiers included in the external critical package status list is also 1, as shown in FIG. Figure 7 As shown, it means that the package in the internal critical package status list and the package in the external critical package status list are the same package, and can be directly regarded as a successful pairing, that is, it is regarded as the same package across the two parts behind the lead curtain. Similar to the case where the quantity is at least two, the internal target tracking package id and the external target tracking package id can be paired and recorded as [idi, idj], for example, recorded as [id1, id2], where the difference between i and j will gradually increase, idi represents the internal target tracking package id, idj represents the external target tracking package id, and the recorded identification pair is stored in the pairing list for subsequent use in package matching and package sorting.

[0083] In this embodiment, when the number of first identifiers included in the internal critical package status list and the number of second identifiers included in the external critical package status list are both one, the matching operation will no longer be performed, but the package corresponding to the fourth package image represented by the second identifier and the package corresponding to the third package image represented by the first identifier are directly determined to be the same package, thereby improving the efficiency of package matching.

[0084] Exemplarily, based on the above embodiments, when matching the third package image in the package tracking box corresponding to the first identifier and the fourth package image in the package tracking box corresponding to each second identifier, the third package image may be input into the feature extraction model to obtain a third feature vector corresponding to the third package image output by the feature extraction model, and each fourth package image may be input into the feature extraction model to obtain a fourth feature vector corresponding to each fourth package image output by the feature extraction model, and a third grayscale distribution histogram corresponding to the third package image may be obtained, and a fourth grayscale distribution histogram corresponding to each fourth package image may be obtained, and then the third package image may be matched with each fourth package image based on the third feature vector, each fourth feature vector, the third grayscale distribution histogram and each fourth grayscale distribution histogram.

[0085] Specifically, when the number of first identifiers included in the internal critical package status list and the number of second identifiers included in the external critical package status list are both at least two, it is necessary to perform another ReID feature vector match, that is, it is necessary to extract the third feature vector corresponding to the third package image and the fourth feature vector corresponding to the fourth package image.

[0086] In the specific implementation process, when the internal and external cameras shoot the same package, there are deviations in shooting angle and light, so it is necessary to extract more feature information for matching. Exemplarily, when the third package image is input into the feature extraction model to obtain the third feature vector corresponding to the third package image output by the feature extraction model, the third package image can be respectively input into the first feature extraction module and the second feature extraction module of the feature extraction model, and the third package image is subjected to feature extraction through the multiple sequentially connected convolutional layers included in the first feature extraction module to obtain the first sub-feature vector output by the first feature extraction module, and the third package image is subjected to feature extraction through the sequentially connected convolutional layers, multi-head attention layers and layerNorm layers in the second feature extraction module to obtain the second sub-feature vector output by the second feature extraction module, and the first sub-feature vector and the second sub-feature vector are subjected to feature fusion to obtain the third feature vector.

[0087] Specifically, since the CNN network has a strong local feature extraction capability, the CNN-based model achieves higher accuracy at a lower cost. However, it focuses on aggregating local features, which hinders its ability to obtain global representation. For this, the ReID task usually uses a multi-head attention mechanism to embed global feature information, but directly embedding global feature information will inevitably lose local feature information. For the actual situation where the package taken between two cameras may be missing, this may be harmful. In view of this, in an embodiment of the present invention, based on the existing resnet50, the structure of the conv-5 layer is improved, so as to fuse the two types of information through the improved feature extraction model. It should be understood that since the multi-head attention mechanism consumes a lot of graphics processing units (GPUs), in order to avoid excessive resource consumption after the improvement, in an embodiment of the present invention, only the conv_5 layer of the original resnet50 model is replaced.

[0088] Fig. 9 It is a schematic diagram of the structure of the conv-5 layer of resnet50 in the prior art. Fig.10 The schematic diagram of the structure of the conv-5 layer of the improved resnet50 provided in the embodiment of the present invention is as follows Fig. 9 and Fig.10As shown, the improved conv-5 layer of resnet50 adds a second feature extraction module based on the existing conv-5 layer, and the second feature extraction module includes a 1*1 convolution layer, a multi-head attention layer and a layerNorm layer connected in sequence. In addition, the improved conv-5 layer of resnet50 also includes a first feature extraction module, and the first feature extraction module includes a 1*1 convolution layer, a 3*3 convolution layer and a 1*1 convolution layer connected in sequence.

[0089] Among them, in order to enable the convolution and multi-head attention mechanism to be integrated, that is, to fuse the results of the first feature extraction module with the results of the second feature extraction module, firstly, 1*1 convolution can be used for the second feature extraction module, and layerNorm is used at the end. Finally, the Fuse structure part of the SKNet network structure is used for the two branches to fuse the two types of information. Among them, the first feature extraction module extracts local feature information through multiple convolutional layers, and the second feature extraction module can extract global feature information through the multi-head attention mechanism.

[0090] SKNet is a deep neural network architecture for image classification and object detection tasks. Its core innovation is the introduction of selective multi-scale convolution kernels and a novel attention mechanism, which improves the ability of feature extraction without increasing the complexity of the network.

[0091] Among them, SKNet mainly consists of three parts: Split, Fuse and Select. In the Split part, the input information X is input into two convolution kernels of different sizes (usually 3x3 and 5x5) to improve efficiency. The Fuse part controls the information flow from different branches through a gating mechanism to fuse information on features of different scales. Finally, in the Select part, the network selects the corresponding features according to the calculated weights.

[0092] SKNet is designed to solve the problem of multi-scale information fusion, so that the network can adapt to features of different scales. Similar to SENet, SKNet also uses an attention mechanism, but the difference is that SKNet's attention mechanism is performed at the convolution kernel level, not at the channel level. This design enables the network to adaptively select appropriate convolution kernels for feature extraction, further improving the performance of the network. In an embodiment of the present invention, the Fuse part used to select appropriate convolution kernels and aggregate convolution kernels can be applied to select the appropriate fusion ratio of convolution branches and multi-head attention branches. Maximize the retention of local feature information and global feature information.

[0093] Therefore, after the third package image is input into the first feature extraction module and the second feature extraction module of the feature extraction model respectively, the first feature extraction module can extract a first sub-feature vector, which is local feature information, and the second feature extraction module can extract a second sub-feature vector, which is global feature information. After the first sub-feature vector and the second sub-feature vector are feature fused, the third feature vector corresponding to the third package image can be obtained.

[0094] After the fourth package image is input into the first feature extraction module and the second feature extraction module of the feature extraction model respectively, the third sub-feature vector can be extracted by the first feature extraction module, and the third sub-feature vector is the local feature information, and the fourth sub-feature vector can be extracted by the second feature extraction module, and the fourth sub-feature vector is the global feature information. After the third sub-feature vector and the fourth sub-feature vector are feature fused, the fourth feature vector corresponding to the fourth package image can be obtained.

[0095] In this embodiment, a multi-head attention layer is added to the feature extraction model, so that the global feature information in the third package image and the fourth package image can be extracted, and the local feature information in the third package image and the fourth package image can be extracted through multiple convolutional layers, so that the extracted information is more comprehensive. When performing package matching through more comprehensive feature information, the matching error problem caused by different shooting angles or light of images taken inside and outside the X-ray machine channel can be avoided, thereby improving the matching accuracy.

[0096] In addition, the third grayscale distribution histogram corresponding to the third package image and the fourth grayscale distribution histogram corresponding to each fourth package image may also be obtained using the tools provided by OpenCV.

[0097] Further, for each fourth eigenvector, the cosine distance between the third eigenvector and the fourth eigenvector can be determined, and the approximate value between the third grayscale distribution histogram and the fourth grayscale distribution histogram corresponding to the fourth eigenvector can be determined, so as to perform weighted summation processing on the cosine distance and the approximate value to obtain the target value, and the fourth package image corresponding to the minimum target value among all the target values ​​is determined as the package image matching the third package image, that is, the package in the fourth package image and the package in the third package image are the same package, and recorded as [idi, idj], wherein idi represents the internal target tracking package id, and idj represents the external target tracking package id, and the recorded identification pair is stored in the pairing list for subsequent use in package matching and package sorting.

[0098] In this embodiment, the packages inside and outside the channel can be matched based on the third eigenvector and the third grayscale distribution histogram of the third package image collected in the channel and the fourth eigenvector and the fourth grayscale distribution histogram of the fourth package image collected outside the channel, thereby improving the accuracy of package matching.

[0099] The package tracking device based on X-ray images and visible light images provided by the present invention is described below. The package tracking device based on X-ray images and visible light images described below and the package tracking method based on X-ray images and visible light images described above can correspond to each other.

[0100] Fig.11 A schematic diagram of a package tracking device based on X-ray images and visible light images provided in an embodiment of the present invention, referring to Fig.11 As shown, the package tracking device 1100 based on X-ray images and visible light images includes:

[0101] The interception module 1101 is used to intercept a first package image including the target contraband from the first visible light image when the target contraband is detected based on the X-ray image; the X-ray image and the first visible light image are taken at the same time;

[0102] A matching module 1102, configured to match the first package image with at least one second package image, where the second package image is a package image within each package tracking frame in at least one second visible light image taken after the first visible light image;

[0103] The tracking module 1103 is used to track the package corresponding to the first package image based on the matching result.

[0104] In an exemplary embodiment, the interception module 1101 is specifically configured to:

[0105] Determining the coordinates of a first center point of a target detection frame representing the target contraband in the X-ray image;

[0106] Determine the coordinates of the second center point of each package detection frame in the first visible light image;

[0107] For each of the second center point coordinates, determining a target distance between the first center point coordinate and the second center point coordinate;

[0108] Intercept the package detection frame to which the second center point coordinates corresponding to the minimum target distance belong;

[0109] The captured image within the package detection frame is determined as the first package image.

[0110] In an exemplary embodiment, the matching module 1102 is specifically configured to:

[0111] Performing feature extraction on the first package image to obtain a first feature vector, and performing feature extraction on each of the second package images to obtain a second feature vector;

[0112] Obtaining a first grayscale distribution histogram corresponding to the first package image, and obtaining a second grayscale distribution histogram corresponding to each of the second package images;

[0113] The first package image is matched with at least one second package image based on the first feature vector, each of the second feature vectors, the first grayscale distribution histogram, and each of the second grayscale distribution histograms.

[0114] In an exemplary embodiment, the matching module 1102 is specifically configured to:

[0115] For each of the second eigenvectors, determining a cosine distance between the first eigenvector and the second eigenvector;

[0116] Determine an approximation between the first grayscale distribution histogram and a second grayscale distribution histogram corresponding to the second eigenvector;

[0117] Determining a target value based on the cosine distance and the approximate value;

[0118] A second package image corresponding to the minimum target value among all the target values ​​is determined as the package image matching the first package image.

[0119] In an exemplary embodiment, the second visible light image is an image taken in an X-ray machine;

[0120] The device further comprises: an acquisition module and a determination module, wherein:

[0121] an acquisition module, configured to acquire a first identifier of each package tracking frame in an internal critical package status list, and acquire a second identifier of each package tracking frame in an external critical package status list; the internal critical package status list includes tracking frames of critical packages that arrive at the lead curtain of an X-ray machine determined based on the second visible light image; the external critical package status list includes tracking frames of critical packages that leave the lead curtain of an X-ray machine determined based on a third visible light image, wherein the third visible light image is an image taken outside the X-ray machine;

[0122] The matching module 1102 is further configured to match, for each of the first identifiers, a third package image in a package tracking frame corresponding to the first identifier with a fourth package image in a package tracking frame corresponding to each of the second identifiers;

[0123] The determination module is used to determine that the package corresponding to the successfully matched fourth package image and the package corresponding to the third package image are the same package.

[0124] In an example embodiment, the apparatus further comprises an adding module, wherein:

[0125] The determination module is further used to determine whether the ordinate of the coordinate of the lower right corner point of each package tracking frame in the second visible light image is equal to a first critical value, where the first critical value is used to represent a critical position of reaching the lead curtain of the X-ray machine;

[0126] The adding module is used to add the package tracking box whose lower right corner point coordinates have a ordinate equal to the first critical value to the internal critical package status list.

[0127] In an exemplary embodiment, the determination module is further used to determine whether the ordinate of the coordinates of the upper left corner point of each package tracking frame in the third visible light image is equal to a second critical value, wherein the second critical value is used to represent a critical position away from the lead curtain of the X-ray machine;

[0128] The adding module is used to add the package tracking box whose upper left corner point coordinates have a ordinate equal to the second critical value to the external critical package status list.

[0129] In an example embodiment, the determination module is used to determine that the package corresponding to the fourth package image represented by the second identifier and the package corresponding to the third package image represented by the first identifier are the same package when the number of first identifiers included in the internal critical package status list and the number of second identifiers included in the external critical package status list are both one.

[0130] In an exemplary embodiment, the matching module 1102 is specifically configured to:

[0131] Inputting the third package image into a feature extraction model to obtain a third feature vector corresponding to the third package image output by the feature extraction model, and inputting each of the fourth package images into a feature extraction model to obtain a fourth feature vector corresponding to each of the fourth package images output by the feature extraction model;

[0132] Obtaining a third grayscale distribution histogram corresponding to the third package image, and obtaining a fourth grayscale distribution histogram corresponding to each of the fourth package images;

[0133] The third package image and each of the fourth package images are matched based on the third eigenvector, each of the fourth eigenvectors, the third grayscale distribution histogram, and each of the fourth grayscale distribution histograms.

[0134] In an exemplary embodiment, the apparatus further comprises: an extraction module and a fusion module, wherein:

[0135] an extraction module, configured to input the third package image into a first feature extraction module and a second feature extraction module of the feature extraction model respectively, perform feature extraction on the third package image through a plurality of sequentially connected convolutional layers included in the first feature extraction module to obtain a first sub-feature vector output by the first feature extraction module, and perform feature extraction on the third package image through a convolutional layer, a multi-head attention layer, and a layerNorm layer sequentially connected in the second feature extraction module to obtain a second sub-feature vector output by the second feature extraction module;

[0136] A fusion module is used to perform feature fusion on the first sub-feature vector and the second sub-feature vector to obtain the third feature vector.

[0137] The device of this embodiment can be used to execute the method of any one of the embodiments of the package tracking method based on X-ray images and visible light images. Its specific implementation process and technical effects are similar to those in the embodiments of the package tracking method based on X-ray images and visible light images. For details, please refer to the detailed description in the embodiments of the package tracking method based on X-ray images and visible light images, which will not be repeated here.

[0138] Fig.12 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.12 As shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230 and a communication bus 1240, wherein the processor 1210, the communication interface 1220 and the memory 1230 communicate with each other through the communication bus 1240. The processor 1210 may call the logic instructions in the memory 1230 to execute a package tracking method based on X-ray images and visible light images, the method comprising: when a target contraband is detected based on the X-ray image, intercepting a first package image including the target contraband from a first visible light image; the X-ray image and the first visible light image are taken at the same time; matching the first package image with at least one second package image, the second package image being a package image in each package tracking frame in at least one second visible light image taken after the first visible light image; and tracking the package corresponding to the first package image based on the matching result.

[0139] In addition, the logic instructions in the above-mentioned memory 1230 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0140] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the package tracking method based on X-ray images and visible light images provided by the above methods, and the method includes: when a target contraband is detected based on an X-ray image, intercepting a first package image including the target contraband from a first visible light image; the X-ray image and the first visible light image are shot at the same time; matching the first package image with at least one second package image, the second package image being a package image within each package tracking box in at least one second visible light image shot after the first visible light image; and tracking the package corresponding to the first package image based on the matching result.

[0141] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the package tracking method based on X-ray images and visible light images provided by the above methods, the method comprising: when a target contraband is detected based on the X-ray image, intercepting a first package image including the target contraband from a first visible light image; the X-ray image and the first visible light image are taken at the same time; matching the first package image with at least one second package image, the second package image being a package image within each package tracking box in at least one second visible light image taken after the first visible light image; and tracking the package corresponding to the first package image based on the matching result.

[0142] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A package tracking method based on X-ray images and visible light images, characterized in that: include: In the case where a target contraband is detected based on the X-ray image, a first package image including the target contraband is intercepted from the first visible light image; The X-ray image and the first visible light image are taken at the same time; matching the first package image with at least one second package image, where the second package image is an image of a package within each package tracking frame in at least one second visible light image taken after the first visible light image; Based on the matching result, tracking the package corresponding to the first package image; The second visible light image is an image taken in an X-ray machine; The method further comprises: Obtaining a first identifier of each package tracking frame in an internal critical package status list, and obtaining a second identifier of each package tracking frame in an external critical package status list; the internal critical package status list includes tracking frames of critical packages that arrive at the lead curtain of the X-ray machine based on the second visible light image; the external critical package status list includes tracking frames of critical packages that leave the lead curtain of the X-ray machine based on the third visible light image, and the third visible light image is an image taken outside the X-ray machine; For each of the first identifiers, matching the third package image in the package tracking frame corresponding to the first identifier with the fourth package image in the package tracking frame corresponding to the second identifier; It is determined that the package corresponding to the successfully matched fourth package image and the package corresponding to the third package image are the same package.

2. The package tracking method based on X-ray images and visible light images according to claim 1, characterized in that: The step of intercepting a first package image including the target contraband from the first visible light image comprises: Determining the coordinates of a first center point of a target detection frame representing the target contraband in the X-ray image; Determine the coordinates of the second center point of each package detection frame in the first visible light image; For each of the second center point coordinates, determining a target distance between the first center point coordinate and the second center point coordinate; Intercept the package detection frame to which the second center point coordinates corresponding to the minimum target distance belong; The captured image within the package detection frame is determined as the first package image.

3. The package tracking method based on X-ray images and visible light images according to claim 1, characterized in that: The matching the first package image with at least one second package image includes: Performing feature extraction on the first package image to obtain a first feature vector, and performing feature extraction on each of the second package images to obtain a second feature vector; Obtaining a first grayscale distribution histogram corresponding to the first package image, and obtaining a second grayscale distribution histogram corresponding to each of the second package images; The first package image is matched with at least one second package image based on the first feature vector, each of the second feature vectors, the first grayscale distribution histogram, and each of the second grayscale distribution histograms.

4. The package tracking method based on X-ray images and visible light images according to claim 3, characterized in that: The matching of the first package image with at least one second package image based on the first feature vector, each of the second feature vectors, the first grayscale distribution histogram, and each of the second grayscale distribution histograms includes: For each of the second eigenvectors, determining a cosine distance between the first eigenvector and the second eigenvector; Determine an approximation between the first grayscale distribution histogram and a second grayscale distribution histogram corresponding to the second eigenvector; Determining a target value based on the cosine distance and the approximate value; A second package image corresponding to the minimum target value among all the target values ​​is determined as the package image matching the first package image.

5. The package tracking method based on X-ray images and visible light images according to claim 1, characterized in that: The method further comprises: Determine whether the ordinate of the lower right corner point coordinate of each package tracking frame in the second visible light image is equal to a first critical value, where the first critical value is used to represent a critical position of reaching the lead curtain of the X-ray machine; The package tracking box whose lower right corner point coordinates have a ordinate equal to the first critical value is added to the internal critical package status list.

6. The package tracking method based on X-ray images and visible light images according to claim 1, characterized in that: The method further comprises: Determine whether the ordinate of the coordinate of the upper left corner point of each package tracking frame in the third visible light image is equal to a second critical value, where the second critical value is used to represent a critical position away from the lead curtain of the X-ray machine; The package tracking box whose upper left corner point coordinates have a ordinate equal to the second critical value is added to the external critical package status list.

7. The package tracking method based on X-ray images and visible light images according to claim 1, characterized in that: The method further comprises: When the number of first identifiers included in the internal critical package status list and the number of second identifiers included in the external critical package status list are both one, it is determined that the package corresponding to the fourth package image represented by the second identifier and the package corresponding to the third package image represented by the first identifier are the same package.

8. The package tracking method based on X-ray images and visible light images according to claim 1, characterized in that: The matching of the third package image in the package tracking frame corresponding to the first identifier and the fourth package image in the package tracking frame corresponding to each of the second identifiers includes: Inputting the third package image into a feature extraction model to obtain a third feature vector corresponding to the third package image output by the feature extraction model, and inputting each of the fourth package images into a feature extraction model to obtain a fourth feature vector corresponding to each of the fourth package images output by the feature extraction model; Obtaining a third grayscale distribution histogram corresponding to the third package image, and obtaining a fourth grayscale distribution histogram corresponding to each of the fourth package images; The third package image and each of the fourth package images are matched based on the third eigenvector, each of the fourth eigenvectors, the third grayscale distribution histogram, and each of the fourth grayscale distribution histograms.

9. The package tracking method based on X-ray images and visible light images according to claim 8, characterized in that: Inputting the third package image into a feature extraction model to obtain a third feature vector corresponding to the third package image output by the feature extraction model includes: Inputting the third package image into the first feature extraction module and the second feature extraction module of the feature extraction model respectively, performing feature extraction on the third package image through a plurality of sequentially connected convolutional layers included in the first feature extraction module to obtain a first sub-feature vector output by the first feature extraction module, and performing feature extraction on the third package image through a sequentially connected convolutional layer, a multi-head attention layer, and a layerNorm layer in the second feature extraction module to obtain a second sub-feature vector output by the second feature extraction module; The first sub-feature vector and the second sub-feature vector are subjected to feature fusion to obtain the third feature vector.

Images