Dual enhanced network method for detecting prohibited goods in long-tail distributed X-ray image

Abstract

The invention provides a dual-enhancement network method for long-tail distribution X-ray image contraband detection. A data-level Poisson fusion enhancement module and a model-level related candidate box aggregation module are included. And the data-level Poisson fusion enhancement module cuts a tail object through a Poisson fusion technology and seamlessly embeds the tail object into a background image, so that diversified tail samples with relatively strong sense of reality are generated, and the long tail distribution problem is effectively relieved by remarkably increasing the number and quality of the tail samples. The model-level related candidate frame aggregation module fuses context information through feature splicing and a full-connection layer to generate relation-enhanced high-order feature representation, so that the model can more accurately distinguish tail-class and head-class objects in a complex scene, and the detection precision of the tail-class objects is remarkably improved. According to the method, the stability of the overall detection performance can be kept while the tail class detection precision can be remarkably improved, and an efficient and robust solution is provided for automatic detection of long-tail distributed objects.

Description

Technical Field

[0001] This invention relates to image detection methods, and in particular to a dual-enhancement network method for detecting contraband in long-tailed X-ray images, belonging to the field of image detection. Background Technology

[0002] X-ray imaging technology is widely used in security inspections to detect contraband in luggage and cargo. However, existing detection algorithms have significant limitations when handling long-tailed data. A long-tailed distribution refers to a dataset where most categories have fewer samples, while a few categories have the vast majority. This imbalance leads to model bias towards common categories (head categories) during training, while the detection or classification accuracy of rare categories (tail categories) drops significantly. Long-tailed distributions are prevalent in many real-world tasks, such as object detection and recommendation systems, posing a major challenge to improving model performance. Researchers have proposed various techniques to address the long-tailed distribution problem, including data resampling, category weighting, and feature enhancement. The long-tailed distribution phenomenon is particularly pronounced in security inspection scenarios, with abundant samples for a small number of head categories (such as common knives and firearms), while samples for most tail categories (such as homemade explosives and special tools) are extremely scarce. This imbalance causes models to favor the detection of head categories, resulting in lower detection performance for tail categories, directly impacting the comprehensiveness and reliability of security inspections.

[0003] Furthermore, X-ray images possess unique visual characteristics, relying primarily on the internal structural information of objects rather than surface texture and color, compared to natural images. This characteristic results in less difference between the target object and the background during detection, leading to blurred features. Additionally, the frequent overlap and occlusion of multiple objects in X-ray images further complicates the model's generalization process.

[0004] In recent years, with the rapid development of deep learning technology, detection models based on convolutional neural networks (CNNs) and Transformers have made significant progress in object detection and classification tasks. However, these models still have limitations when dealing with the characteristics of X-ray images and long-tailed data distribution. On the one hand, they are highly dependent on the data and cannot effectively solve the problem of imbalanced samples; on the other hand, they do not make sufficient use of contextual information and have difficulty accurately classifying occluded or similar-looking targets.

[0005] With increasing attention from academia and industry to the problem of long-tailed distribution detection, some studies have begun to attempt to improve existing technologies and have proposed data-level and model-level solutions, but these have not yet fully solved the practical problems in X-ray detection tasks.

[0006] To address the above shortcomings, this invention proposes a dual-enhancement network method for contraband detection in long-tailed X-ray images. By combining data-level and model-level enhancements, the method significantly improves the overall performance of long-tailed X-ray image detection tasks, providing a more efficient and reliable solution for practical applications such as security checks. Summary of the Invention

[0007] This invention presents a dual-enhancement network method for contraband detection in long-tailed X-ray images, employing both data-level and model-level enhancement optimization. For data-level enhancement, Poisson fusion technology is used to generate realistic tail-class samples, significantly improving data quality and diversity. Poisson fusion is an image processing-based synthesis technique designed to seamlessly embed cropped objects into the background image, maintaining visual consistency. By addressing issues such as lighting differences and texture breaks between the inserted object and the background, this technique generates more realistic and natural synthetic images. Poisson fusion has wide applications in image data enhancement and object detection data augmentation, particularly in generating samples for long-tailed categories, where it can alleviate data imbalance. For model-level enhancement, a context-aware feature enhancement module is introduced, utilizing the spatial and semantic relationships between the target object and its environment to optimize classification capabilities. This comprehensive solution significantly improves the overall performance of long-tailed X-ray image detection tasks.

[0008] This invention discloses a dual-enhancement network method for detecting contraband in long-tailed X-ray images, comprising two modules: a data-level Poisson fusion enhancement module and a model-level relevance candidate box aggregation module. Data-level enhancement generates realistic synthetic samples by cropping tail-class objects and seamlessly inserting them into other images using Poisson fusion technology, thus mitigating the data imbalance problem. Model-level enhancement constructs context-aware features, utilizing the spatial and semantic relationships between the target object and its surrounding environment to optimize classification performance, making it particularly suitable for complex occlusion scenarios. The combination of these two modules significantly improves tail-class detection accuracy while maintaining overall detection performance stability, providing an efficient and robust solution for the automatic detection of long-tailed objects. The technical details of the two modules are as follows:

[0009] Data-level Poisson fusion enhancement module:

[0010] The data-level Poisson fusion enhancement module aims to address the severe imbalance between head and tail classes in long-tail data distributions by improving the representation capabilities of tail classes through data augmentation. The specific implementation process includes two steps: instance pruning and Poisson fusion.

[0011] First, for instances of the tail category, the corresponding target region is cropped from images containing that category. A manually labeled segmentation mask ensures the cropped region accurately includes the foreground object, avoiding the introduction of unnecessary background information. Next, the cropped foreground instances are inserted into different background images, prioritizing metal surfaces for image fusion. This is because overlapping areas between non-metallic and metallic objects are more difficult to detect in X-ray scenes. This approach increases the diversity of training in complex scenes, further improving the model's ability to distinguish overlapping objects. Simultaneously, Poisson image editing techniques are used to naturally blend the foreground and background, ensuring the inserted object and the new background are consistent in lighting and texture, generating visually natural synthetic images. Finally, these synthetic images, along with updated bounding boxes and category labels, are added back to the training dataset, effectively increasing the number of tail class samples and alleviating data imbalance.

[0012] This enhancement method improves the model's learning ability for tail classes by balancing the effective sample size of each class, and specifically optimizes the detection performance of overlapping areas of metal and non-metal in X-ray detection scenarios, thereby improving the model's generalization ability and detection effect.

[0013] Model-level related candidate box aggregation module:

[0014] The model-level related candidate box aggregation module aims to improve the model's ability to distinguish between head and tail classes by utilizing contextual information from surrounding objects, especially in the detection of simple and easily confused metal tools in real-world scenarios. Unlike data-level augmentation, which focuses on balancing the number of training samples, model-level augmentation improves feature representation by integrating spatial relationships.

[0015] The specific process is as follows: First, from the proposals generated by the detection head, the top k high-confidence proposals are selected based on their confidence scores. These proposals represent the most salient prohibited items in the image, providing reliable features for relation inference. For the selected top k proposals, their feature vectors are concatenated to form a single feature representation, capturing the relationships and dependencies between these high-confidence proposals, thereby encoding contextual information reflecting interactions between items. Next, the concatenated features are processed through a fully connected layer to generate a fused relation feature. This feature further integrates the contextual information of all selected salient objects and improves the ability to represent higher-order relations.

[0016] Subsequently, the fused relational features are concatenated with the feature vectors of each original proposal to form an enhanced feature representation. To ensure that the dimensions of the enhanced features are consistent with those of the original proposal features, a second fully connected layer is used to process the concatenated features, resulting in an updated feature vector. This step ensures that the new features can seamlessly replace the original features and be integrated into the subsequent detection process. Whether to use an activation function in this process depends on the specific settings during detection head feature extraction and score generation, and it is necessary to ensure that the distribution of the new features is consistent with that of the original features.

[0017] Finally, the updated features replace the original proposed features, the classification logits are recalculated, and used as input to the classification and bounding box regression heads. By introducing enhanced features containing contextual information, this process improves the performance of the detection model in complex or occluded scenes, especially the detection capability of tail classes, effectively solving the problem of object confusion.

[0018] This invention discloses a dual-enhancement network method for detecting contraband in long-tailed X-ray images, specifically comprising the following steps:

[0019] First, the input X-ray image is image data containing multiple categories, mainly involving head and tail class instances in long-tailed distribution scenes. The input image includes bounding boxes and category label information, which are used for subsequent instance cropping and enhancement operations.

[0020] Then, for instances of the tail category, the corresponding target region is cropped from the input image. A manually labeled segmentation mask ensures that the cropped region accurately includes the foreground object, avoiding the introduction of unnecessary background information and thus generating a clean foreground instance.

[0021] Subsequently, the cropped foreground instances are inserted into different background images. Particularly in X-ray scenes, metallic surfaces are prioritized for fusion to increase the complexity of overlapping non-metallic and metallic objects. Poisson image editing techniques are employed during the fusion process to ensure natural blending of the foreground and background in terms of lighting and texture, generating a visually consistent composite image.

[0022] Next, feature extraction is performed on the enhanced image input, and candidate proposals and corresponding feature vectors are generated through the model's detection head. These feature vectors contain spatial information and contextual relationships of objects, providing a foundation for subsequent feature aggregation.

[0023] During feature aggregation, high-confidence proposals from the detection head are selected. Contextual information is then fused through feature concatenation and a fully connected layer to generate higher-order feature representations with enhanced relationships. Subsequently, the enhanced features are concatenated and reconstructed with the original proposal features to improve the model's ability to represent tail-like objects and complex scenes.

[0024] The updated features are then used for classification and bounding box regression to generate the final detection results. In this process, the enhanced features more effectively address the issues of insufficient tail class detection performance and easy confusion between objects.

[0025] Finally, through end-to-end training, the data-level augmentation and model-level feature optimization processes are jointly optimized until the model's detection results meet the expected performance requirements or the training iterations are reached. During training, the model gradually learns the feature representations of tail classes and object relationships in complex scenes, improving detection performance and generalization ability.

[0026] As can be seen from the above technical solutions, the present invention has the following advantages:

[0027] 1. Compared with existing data augmentation methods, the data-level Poisson fusion augmentation method proposed in this invention ensures the accuracy of foreground instance clipping through manually labeled segmentation masks, and prioritizes inserting tail class instances into the surface of metal categories for fusion in X-ray scenes. This effectively solves the problem of difficult detection of overlapping metal and non-metal objects, significantly improves the detection performance of tail classes in complex backgrounds, and alleviates the class imbalance problem in long-tail data distribution.

[0028] 2. Compared with traditional model augmentation methods, the model-level relevance proposal aggregation strategy of this invention selects high-confidence proposals and extracts their features for contextual relationship fusion, encoding spatial dependencies between salient objects and further enhancing object feature representation. Through feature reconstruction and feedback, it effectively solves the problem of recognizing simple-shaped and easily confused metal tools in complex scenes, especially significantly improving the detection performance of tail-like objects. Attached Figure Description

[0029] Figure 1 The diagram of the dual-enhancement network model of this invention.

[0030] Figure 2 The flowchart of the dual-enhancement network of the present invention. Detailed Implementation

[0031] The invention will now be described in more detail with reference to the accompanying drawings.

[0032] This invention discloses a dual-enhancement network method for contraband detection in long-tailed X-ray images, comprising two modules: a data-level Poisson fusion enhancement module and a model-level correlation candidate box aggregation module. Please refer to [link / reference]. Figure 1 This is a diagram of the dual-enhancement network model of the present invention.

[0033] Among them, the data-level Poisson fusion enhancement module:

[0034] First, for instances of the tail category, the corresponding target region is cropped from the image containing that category, and a manually labeled segmentation mask is used to ensure that the cropped region accurately contains the foreground object, avoiding the introduction of unnecessary background information.

[0035] Next, the cropped foreground instances are inserted into different background images, especially prioritizing metal surfaces for image fusion, because in X-ray scenes, the overlapping areas of non-metallic and metallic objects are more difficult to detect. This method can increase the diversity of training in complex scenes and further improve the model's ability to distinguish overlapping objects.

[0036] At the same time, the foreground and background are naturally blended through Poisson image editing technology, ensuring that the inserted object and the new background are consistent in lighting and texture, generating a visually natural composite image.

[0037] Finally, these synthesized images, along with the updated bounding boxes and class labels, are added back to the training dataset, effectively increasing the number of samples in the tail class and alleviating the data imbalance problem.

[0038] Among them, the model-level related candidate box aggregation module:

[0039] First, from the proposals generated by the detection head, the top k high-confidence proposals are selected based on their confidence scores. These proposals represent the most salient prohibited items in the image, providing reliable features for relation inference. The feature vectors of the selected top k proposals are concatenated to form a single feature representation, capturing the relationships and dependencies between these high-confidence proposals, thus encoding contextual information reflecting interactions between items. Next, the concatenated features are processed through a fully connected layer to generate a fused relation feature. This feature further integrates the contextual information of all selected salient objects and enhances the ability to represent higher-order relations.

[0040] Subsequently, the fused relational features are concatenated with the feature vectors of each original proposal to form an enhanced feature representation. To ensure that the dimensions of the enhanced features are consistent with those of the original proposal features, a second fully connected layer is used to process the concatenated features, resulting in an updated feature vector. This step ensures that the new features can seamlessly replace the original features and be integrated into the subsequent detection process. Whether to use an activation function in this process depends on the specific settings during detection head feature extraction and score generation, and it is necessary to ensure that the distribution of the new features is consistent with that of the original features.

[0041] Finally, the updated features replace the original proposed features, the classification logits are recalculated, and used as input to the classification and bounding box regression heads. By introducing enhanced features containing contextual information, this process improves the performance of the detection model in complex or occluded scenes, especially the detection capability of tail classes, effectively solving the problem of object confusion.

[0042] This invention discloses a dual-enhancement network method for contraband detection in long-tailed X-ray images, comprising seven steps. Please refer to [link / reference]. Figure 2 This is a flowchart of the dual-enhancement network of the present invention. The seven steps are as follows:

[0043] S1: Data Input. The input consists of image data containing multiple categories, primarily involving head and tail class instances in long-tailed distribution scenarios. The input image includes bounding boxes and category label information for subsequent instance cropping and enhancement operations.

[0044] S2: Instance cropping. For tail-category instances, the corresponding target region is cropped from the input image. A manually labeled segmentation mask is used to ensure that the cropped region accurately contains the foreground object, avoiding the introduction of unnecessary background information, thereby generating a clean foreground instance.

[0045] S3: Poisson blending. The cropped foreground instance is inserted into different background images. Especially in X-ray scenes, metallic surfaces are preferentially selected for blending to increase the complexity of overlapping non-metallic and metallic objects. Poisson image editing techniques are used during the blending process to ensure that the foreground and background blend naturally in terms of lighting and texture, generating a visually consistent composite image.

[0046] S4: Feature extraction. Feature extraction is performed on the enhanced input image. Candidate proposals and corresponding feature vectors are generated through the model's detection head. These feature vectors contain spatial information and contextual relationships of objects, providing a foundation for subsequent feature aggregation.

[0047] S5: Feature aggregation selects high-confidence proposals from the detector head output, and generates high-order feature representations with enhanced relationships by fusing contextual information through feature concatenation and fully connected layers. Subsequently, the enhanced features are concatenated and reconstructed with the original proposal features to improve the model's ability to express tail-like objects and complex scenes.

[0048] S6: Output the results. Use the updated features for classification and bounding box regression to generate the final detection results. In this process, the enhanced features can more effectively solve the problems of insufficient tail class detection performance and easy confusion between objects.

[0049] S7: Model training involves end-to-end training, jointly optimizing data-level augmentation and model-level feature optimization until the model's detection results meet expected performance requirements or the required number of training iterations is reached. During training, the model gradually learns tail-class feature representations and object relationships in complex scenes, improving detection performance and generalization ability.

[0050] To demonstrate the technical superiority of the method of this invention, we evaluated the proposed method on two publicly available large-scale X-ray datasets, HIXray and PIDray.

[0051] Table 1 shows the test results of the dual-enhancement network method for contraband detection in long-tailed X-ray images proposed in this invention on the X-ray datasets HIXray and PIDray. Here, F-RCNN represents the baseline method Faster R-CNN, and +ours indicates the performance of the baseline method after adding our method. S-RCNN represents Sparse R-CNN, and the other baseline methods are referred to by their full names: RetinaNet, CenterNet, and ATSS. As can be seen from Table 1, overall, the above five baseline methods all exhibited superior performance after adding our method, especially for the tail-type NL category, which we focused on improving, resulting in a significant improvement in detection accuracy.

[0052] Table 1. Test results of the method of the present invention on the datasets HIXray and PIDray.

[0053]

[0054] Table 2 shows the full names of the actual categories corresponding to the abbreviations in Table 1.

[0055] Table 2 lists the full names of the actual categories corresponding to the abbreviations in Table 1.

[0056]

Claims

1. A dual-enhancement network method for contraband detection in long-tailed X-ray images, characterized in that, Includes the following steps: S1: Data input, which is image data containing multiple categories, mainly involving head and tail class instances in long-tail distribution scenarios; The input image contains bounding box and category label information, which is used for subsequent instance cropping and enhancement operations; S2: Instance cropping. For tail-category instances, the corresponding target region is cropped from the input image. The cropped region is ensured to accurately contain the foreground object by manually annotated segmentation mask, avoiding the introduction of unnecessary background information, thereby generating a clean foreground instance. S3: Poisson fusion, the cropped foreground instance is inserted into different background images. Especially in X-ray scenes, metal surfaces are preferred for fusion to increase the complexity of non-metallic and metallic object overlap. The fusion process employs Poisson image editing technology to ensure that the foreground and background blend naturally in terms of lighting and texture, generating a visually consistent composite image; S4: Feature extraction, which extracts features from the enhanced input image. The model's detection head generates candidate proposals and corresponding feature vectors. These feature vectors contain spatial information and contextual relationships of objects, providing a foundation for subsequent feature aggregation. S5: Feature aggregation: Select high-confidence proposals from the detector head output, and generate high-order feature representations with enhanced relationships by fusing contextual information through feature concatenation and fully connected layers. Subsequently, the enhanced features are concatenated and reconstructed with the original proposal features to improve the model's ability to express tail-like objects and complex scenes. S6: Output the results. Use the updated features for classification and bounding box regression to generate the final detection results. In this process, the enhanced features can more effectively solve the problems of insufficient tail class detection performance and easy confusion between objects. S7: Model training, through end-to-end training, jointly optimizes the data-level augmentation and model-level feature optimization processes until the model's detection results meet the expected performance requirements or the training iterations are reached; During training, the model gradually learns the feature representations of tail classes and the relationships between objects in complex scenes, thereby improving detection performance and generalization ability.

2. The dual-enhancement network method for contraband detection in long-tailed X-ray images according to claim 1, characterized in that, In step S5, the top k high-confidence proposals generated by the detection head are selected based on their confidence scores. These proposals represent the most salient prohibited items in the image and provide reliable features for relational reasoning. For the selected top k proposals, their feature vectors are concatenated to form a single feature representation that captures the relationships and dependencies between these high-confidence proposals, thereby encoding contextual information reflecting interactions between items. Then, the concatenated features are processed through a fully connected layer to generate a fused relational feature. This feature further integrates the contextual information of all selected salient objects and improves the ability to represent higher-order relations.

Images