Intra-domain and inter-domain object learning method for point supervision X-ray contraband detection

Through the point-supervised intra-domain object-based learning method, combined with local Gaussian occlusion and wavelet decoupling adversarial learning, the problem of high cost of frame annotation and inter-domain differences in object detection in X-ray pictures is solved, and efficient and flexible Improved detection performance.

CN119090818AActive Publication Date: 2024-12-06XIAMEN UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411089623.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-06
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

The prior art prohibits object detection in X-ray pictures to rely on box annotation, which is costly and inefficient, and the domain adaptive object detection method has poor performance when the difference between natural pictures and X-ray pictures is large.

Method used

The intra-interdomain object learning method is adopted with point-supervised intradomain object learning method, and the intradomain object information is mined through local focus Gaussian occlusion and global random Gaussian occlusion, and the object information in natural pictures is transferred to the wavelet decoupling adversarial learning and objectivity modules. In X-ray pictures, reduce modal differences and improve detection performance.

Benefits of technology

It significantly reduces the annotation cost, improves the performance and generalization ability of prohibited item detection in X-ray pictures, alleviates the local dominant problems caused by severe intra-class changes, and achieves excellent performance on multiple X-ray data sets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090818A_ABST
    Figure CN119090818A_ABST
Patent Text Reader

Abstract

The invention discloses an intra-domain-inter-domain object learning method for point supervision X-ray prohibited article detection, and the method carries out the X-ray prohibited article detection under point supervision through an intra-domain-inter-domain object module, and is composed of two key modules: an intra-domain object learning module and an inter-domain object learning module. The intra-domain object learning module designs a local focus Gaussian masking block and a global random Gaussian masking block to jointly learn the object in the X-ray picture. The inter-domain object learning module introduces an adversarial learning block and an object block based on wavelet decomposition, so that the modal difference is effectively reduced, and object knowledge learned from a natural picture with instance-level labels is migrated to an X-ray picture. According to the scheme, the problem of local dominance caused by serious intra-class changes in the X-ray picture is relieved. Experimental results on four X-ray data sets show that the annotation cost is remarkably reduced, meanwhile, excellent performance is achieved, and the practicability is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an intra-domain and inter-domain objectivity learning method for point-supervised X-ray contraband detection. Background Art

[0002] X-ray images are characterized by complex backgrounds, severe occlusions, limited color and detail information, and often lack of contour details, which may affect the extraction of overall information about objects. However, fully supervised object detection methods can obtain overall information from instance-level annotations and the X-ray images themselves, thereby achieving good performance. When it comes to improving the performance of fully supervised object detection, since instance-level annotations are fixed for a specific dataset, the overall information they can provide is relatively fixed. Therefore, most existing methods focus on mining the inherent overall information of X-ray images. It is safe to say that the good performance of these methods is also based on a large amount of expensive instance-level supervision.

[0003] As a special type of object detection task, X-ray prohibited object detection aims to detect the location and category of prohibited objects in X-ray images. Existing prohibited object detection methods are usually based on popular object detection paradigms (including one-stage detectors and two-stage detectors). Different from object detection in natural images, prohibited object detection in X-ray images is usually affected by severe occlusion and object overlap. Some methods have been developed to address the problem of object occlusion in X-ray images, for example, Wei et al. developed a De-Occlusion Attention Module (DOAM) containing an edge attention module and a region attention module to detect occluded objects. Miao et al. assumed that each input image is sampled from a mixed distribution and introduced a class-balanced hierarchical refinement method (CHR). Tao et al. adopted a local inhibition module (LIM) to identify discriminative features while ignoring irrelevant information in occluded objects. Wang et al. proposed a Selective Dense Attention Network (SDANet).

[0004] All of the above methods rely heavily on box annotations. Although this approach can bring good performance, it usually requires a lot of manpower to collect accurate annotations. In order to balance the annotation cost and detection performance, this scheme studies forbidden object detection under point supervision, which only provides an anchor point to locate the area near the center of the object, thereby greatly reducing the total annotation time.

[0005] Weakly supervised object detection methods do not use box annotations, but instead use weaker forms of supervision (such as image supervision, point supervision) to reduce the annotation cost. Many methods have been proposed, such as WSDDN, which first introduced multiple instance learning (MIL) to weakly supervised object detection. OICR alleviates the partial dominance problem through iterative optimization. PCL expands the detection region through proposal clustering. ICMWSD solves the partial dominance problem by adopting a multi-instance self-training method and masks, while reducing memory consumption through batch backpropagation. OD-WSCL uses contrastive learning to obtain better pseudo-label accuracy. Basically, these methods rely on prior box generation to propose regions based on the overall information of the image, and then use MIL to filter these proposals. However, these prior box generation methods are ineffective on X-rays. Point-supervised object detection methods effectively solve the problems of too many or insufficient instances and high memory consumption. In this paper, a simple but effective wavelet decomposition-based method is studied to transfer the object information learned from natural images to X-ray images, which clearly demonstrates the great potential of using natural images to solve the problem of prohibited object detection in X-ray images.

[0006] Domain adaptive object detection aims to transfer the model trained on the source domain to the target domain for testing. Existing methods can be roughly divided into three categories. The first category of methods focuses on adjusting the feature distribution of different domains by applying adversarial learning or minimizing the maximum mean difference between the source and target domains. The second category of methods adopts a self-training strategy, aiming to generate high-quality pseudo labels in the target domain. The third category of methods uses a teacher-student framework to achieve domain adaptive detection through the consistency constraints of detector predictions. In the first category of methods, Saito et al. proposed an adaptive detector method based on strong local alignment and weak global alignment. The second category of methods adopts a self-training strategy, aiming to generate high-quality pseudo labels in the target domain. The third category of methods uses a teacher-student framework to achieve domain adaptive detection through the consistency constraints of detector predictions.

[0007] These methods are usually used in domains with small gaps and similar categories, but the performance of these methods in X-ray images is not satisfactory due to the large differences between natural images and X-ray images. Therefore, this application studies a method to transfer the target information learned from natural images to X-ray images based on a simple but effective wavelet decomposition. This clearly shows the great potential of using natural images to solve the detection of prohibited objects in X-ray images. Summary of the invention

[0008] In view of this, the purpose of the present invention is to propose an intra-domain and inter-domain object learning method for point-supervised X-ray contraband detection, which is reliable in implementation, flexible in application and can alleviate the problem of local dominance caused by severe intra-class variations in X-ray image detection.

[0009] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0010] A method for intra-domain and inter-domain objectivity learning for point-supervised X-ray contraband detection, comprising:

[0011] A. Prepare a dataset of X-ray prohibited items and a dataset of natural images, where the former only contains annotation points and the latter contains instance-level annotations;

[0012] B. Input the X-ray prohibited items dataset and the natural image dataset into the feature extractor to obtain the feature maps corresponding to the two datasets respectively;

[0013] C. Design an in-domain object learning network, which includes local focal Gaussian masking and global random Gaussian masking to mine in-domain object information;

[0014] D. Input the feature map of the X-ray prohibited items dataset obtained in step B into the in-domain object learning network to obtain an enhanced feature map, and then send it to the multi-instance branch learning network to predict the instance score of each instance generated around its annotation point;

[0015] E. Design an inter-domain object learning network, which includes a wavelet decoupled adversarial learning network and an object module;

[0016] F. Perform wavelet decoupling on the feature graph of the X-ray prohibited items dataset and the feature graph of the natural image dataset obtained in step B using a wavelet decoupling adversarial learning network, and perform adversarial learning on the low-frequency style features;

[0017] G. Use the objectness module to predict the objectness score of each target in the feature map of the X-ray prohibited items dataset and the natural image dataset;

[0018] H. Weight the objectness score of the feature graph of the X-ray prohibited items dataset and the instance score obtained by processing in step D to obtain the final instance score of the multi-instance branch learning network; for the natural image dataset, its objectness score is supervised by instance-level annotation;

[0019] I. Select instances whose instance scores of the X-ray prohibited items dataset are higher than the preset value, use them as pseudo labels, and then train a fully supervised prohibited items detection model.

[0020] As a possible implementation, further, in step A of this solution, the X-ray prohibited items dataset includes an OPIXray dataset, a HIXray dataset, a SIXray dataset, and a PIDray dataset; the OPIXray dataset contains 5 categories, totaling 8,885 images; the SIXray dataset contains 6 categories, totaling 1,059,231 images; the HIXray dataset contains 8 categories, totaling 45,364 images; the PIDray dataset contains 12 categories, totaling 124,486 images;

[0021] Among them, the OPIXray dataset, HIXray dataset, SIXray dataset, and PIDray dataset all include instance-level annotations. This scheme uses 47,677 images containing contraband for training and testing. Points are randomly generated according to Gaussian distribution near the center point of the instance-level annotation as the annotation points used in the training process;

[0022] The natural image dataset is the commonly used COCO14 dataset; in the COCO14 dataset, the training set has 82,783 images, the validation set has 40,504 images, and the test set has 40,775 images; this scheme randomly selects the same number of images as the X-ray dataset from the COCO14 training set to alleviate the imbalance problem in the number of images in the X-ray prohibited items dataset and the natural image dataset.

[0023] As a preferred implementation option, preferably, in step B of this solution, the feature extractor is constructed by combining ResNet50 and FPN (feature pyramid network); ResNet50 and FPN (feature pyramid network) are combined to form a multi-level feature extraction network; ResNet50 is responsible for extracting more detailed and local features at the bottom layer, and FPN performs feature fusion and introduction of context information at the high layer;

[0024] Among them, ResNet50 is a classic convolutional neural network structure with strong feature extraction capabilities, while FPN (Feature Pyramid Network) can extract and fuse features at different scales, allowing the network to better handle objects of different sizes.

[0025] By combining the feature extractor built with ResNet50 and FPN, the features related to prohibited items are extracted from the X-ray prohibited items dataset. At the same time, the features of general objects are extracted from the natural image dataset. The feature maps corresponding to the two datasets are obtained, which are set as natural images and X-ray images respectively, to obtain feature representations for different datasets. This combination can enable the network to have better receptive field and semantic information, thereby improving the performance of prohibited items detection; at the same time, feature representations for different datasets can be obtained, providing richer and more accurate information for subsequent prohibited items detection tasks.

[0026] As a preferred implementation option, preferably, step C of this solution includes:

[0027] The in-domain object learning network is designed, which includes Local Focus Gaussian Masking (LFGM) and Global Random Gaussian Masking (GRGM) to mine the in-domain object information; the purpose of designing the in-domain object learning network is to better mine and utilize the in-domain object information. By combining LFGM and GGRM, the network can improve the robustness, generalization ability and recognition accuracy of the model, thereby achieving more accurate and reliable detection and recognition of prohibited items in complex scenes.

[0028] Among them, the LFGM and GRGM blocks are designed to better utilize the object information in the domain. The local focus Gaussian mask LFGML guides the model to focus on the non-central area of ​​the prohibited items by generating a local Gaussian mask; by finding the point with the local maximum activation value and generating a Gaussian mask around the point, the model is explicitly forced to learn the non-central area of ​​the prohibited items; the advantage of this is that the model no longer pays too much attention to the area near the marked point, but can more comprehensively understand the shape and structure of the entire prohibited item. Through the LFGM block, the model is able to learn and utilize various parts of the prohibited items to improve the accuracy of detection and recognition.

[0029] Specifically, considering that the proposal boxes are generated based on the point annotations around the center of the prohibited items, almost all proposal boxes include the annotated central area of ​​the prohibited items. For example, when the model is trained to detect hammers, the proposal boxes are generated around the annotation points. This approach may cause the model to focus on the handles around the annotation points and ignore other non-central discriminative areas (such as the head). To address this issue, we search for points with local maximum activation values ​​and generate local Gaussian masks to explicitly force the model to learn the non-central areas of the prohibited items.

[0030] Technically, it includes: for a given X-ray image In and its corresponding point annotations First, take the picture I n Input backbone network and FPN to obtain feature maps of different scales in L represents the number of feature maps, H l , W l and C l Represent the feature map F n,l The height, width and number of channels of the feature map F n,l Each channel of l )×(βH l ) finds the point with the highest activation in the window, β represents the scaling factor, and this point is used as the average position for generating the local Gaussian mask; the above process can be expressed as:

[0031]

[0032] in, Represents the c-th channel feature map The average position on

[0033] At the same time, for each F n,l Define a covariance matrix ∑ l =αR l ,in is a two-dimensional diagonal matrix, α represents the scaling factor;

[0034] Finally, a local Gaussian mask is applied to each channel of the feature map,

[0035]

[0036] in, represents the feature map after masking, represents a local Gaussian mask;

[0037] Global Random Gaussian Masking GRGM enhances the robustness and generalization of the model by introducing global random Gaussian masking. The GRGM block randomly masks certain areas of the image, forcing the model to learn and rely on other recognizable features. Since the masking is randomly generated, the model needs to learn features outside the masked area to predict the target, thereby enhancing the model's ability to recognize the shapes and locations of different prohibited items. Through the GRGM block, the model can better adapt to various interferences and changes, and improve performance and generalization in complex environments.

[0038] By applying the LFGM block, the model is forced to learn non-central areas. However, the model may still focus on local areas; by randomly applying multiple Gaussian masks on the feature map through the GRGM block, the model is prevented from over-focusing on the local discriminative areas and globally learning the entire object, which includes the central and non-central areas; this method is conducive to utilizing the object knowledge of the object's internal modality. Unlike the LFGM block, the GRGM block randomly generates points in each feature map. Specifically, for each feature map F n,l , randomly generate a set of points in Represents the feature map F n,l The mth randomly generated point (x coordinate and y coordinate) on the graph, where M represents the total number of points generated; based on this, for each point Generate a Gaussian mask

[0039] Apply these Gaussian masks to the feature map F n,l , and subtract them from the original feature map,

[0040]

[0041] Where F″ n,l Represents the feature map after masking;

[0042] In obtaining F′ n,l and F″ n,l Finally, the two feature maps are concatenated and a 1×1 convolutional layer is used to reduce their channel number to obtain feature E n,l , which is used as the input feature of the MIL branch.

[0043] As a preferred implementation option, preferably, step E of this solution includes:

[0044] Design an inter-domain objectivity learning network to effectively transfer objectivity information extracted from natural images to X-ray images; the network consists of two key components, including Wavelet Decomposition Based Adversarial Learning (WDAL) and Objectness Block (OB);

[0045] Among them, the wavelet decoupling adversarial learning network uses wavelet decomposition technology to decouple the feature graphs of the X-ray prohibited items dataset and the natural image dataset, dividing them into two parts: style and content; then, through adversarial learning, the modal differences caused by style differences are reduced, so that the model can better adapt to the characteristics of X-ray images;

[0046] The objectivity module uses the transferred objectivity knowledge and integrates it into the multi-instance learning branch. By calculating the objectivity score of each region of interest RoI feature, the module can enhance the probability that the proposal box contains the entire prohibited item. These two components work together to enable the model to accurately identify and locate prohibited items in X-ray images, improving the efficiency and accuracy of security detection.

[0047] As a preferred implementation option, preferably, step F of this solution includes:

[0048] In the wavelet decomposition adversarial learning network, it is not ideal to directly transfer the object knowledge extracted from natural images to X-ray images due to the significant modal differences between natural images and X-ray images. To solve this problem, the feature maps of the X-ray prohibited items dataset and the feature maps of the natural image dataset are decoupled into two parts: style and content through the WDAL module using wavelet decomposition, and then the modal differences caused by style differences are reduced through adversarial learning; the WDAL module is inspired by the effectiveness of wavelet decomposition in preserving content information.

[0049] Technically, Haar wavelet is used for wavelet decomposition. Haar wavelet includes four kernels, namely {LL T , L.H. T , H.L. T , HH T}, where L and H are low-pass and high-pass filters respectively, defined as follows:

[0050]

[0051] Wavelet decomposition transforms the feature map M n,l , which is from the X-ray image I n or natural picturesJ n Extract it and decompose it into four parts, namely

[0052] A n,l =M n,l *(LL T )

[0053] H n,l =M n,l *(LH T )

[0054] V n,l =M n,l *(HL T )

[0055] D n,l =M n,l *(HH T )

[0056] Where * represents the convolution operation; A n,l Indicates the low-frequency part; H n,l 、V n,l and D n,l Represents the high frequency part;

[0057] Among them, the low-frequency part mainly corresponds to the style features, capturing the detailed style information of the image; the high-frequency part corresponds to the content features, focusing on local details and edges; then, A n,l Input into a convolutional layer with learnable parameters to obtain the converted style feature A′ n,l ; In order to reduce the modality difference, adversarial learning is performed on the converted style features, and the natural image dataset and the X-ray prohibited items dataset are regarded as the source domain and the target domain respectively; a domain classifier is trained on the activation value of each converted style feature to predict the modality of each style feature; in mathematical principle, let is the domain label of the nth image, where if the nth image belongs to the source domain, then Otherwise, it is 1; the activation value of the converted style feature at the coordinate (x, y) is expressed as Using corresponding style features The predicted value for each position Calculate the cross-entropy based adversarial loss L adv ,Right now

[0058]

[0059] To align the modal distributions, we simultaneously optimize the parameters of the modal classifier, by minimizing the adversarial loss, and the parameters of the base network, by maximizing this loss. We also use a gradient reversal layer GRL to train the modal classifier.

[0060] Then, the content feature H n,l 、V n,l and D n,l and the converted style feature A′ n,l Connect and get the feature H without modality specificity n,l ,Right now

[0061]

[0062] in Represents a concatenation operation.

[0063] As a preferred implementation option, preferably, step G of this solution includes:

[0064] The objectivity module is used to predict the objectivity score of each target in the feature map of the X-ray prohibited items dataset and the natural image dataset. The objectivity information extracted from the natural image is transferred to the X-ray image using the WDAL module. This information is then integrated into the MIL branch of the objectivity module to enhance the probability that the proposal box contains the entire prohibited item.

[0065] The objectness score reflects the probability that each region of interest RoI feature contains the entire object; it includes: represents the proposed RoI feature generated by the kth annotation point of the nth X-ray prohibited items image or natural image, where R represents the total number of proposals;

[0066] RoI pooling is performed on each feature map, and the final objectness score is calculated through a fully connected layer. The objectness module can be expressed as:

[0067]

[0068] Among them, RoI_Pooling() and FC() represent RoI pooling and fully connected layers respectively; Represents the objectness score corresponding to the r-th proposal box generated by the k-th point;

[0069] For natural images, rich box annotations can be used to train the objectness module. Specifically, for the proposal boxes generated from the point annotations, the intersection-over-union ratio between them and the annotated box annotations is calculated. Among them, the proposal box with an intersection-over-union ratio greater than or equal to 0.5 is regarded as a positive sample and marked as 1, and the proposal box with an intersection-over-union ratio less than this value is regarded as a negative sample and marked as 0. Based on this, the cross entropy loss function can be used to adjust the model parameters, that is,

[0070]

[0071] in, and They represent the area annotated by the r-th proposal box and its corresponding annotation box respectively;

[0072] Finally, I 2 The joint loss of OL-Net is

[0073] L=L mil +λ 1 L adv +λ 2 L obj

[0074] Among them, L mil represents the total MIL loss defined in P2BNet; λ 1 and λ 2 is the balance weight.

[0075] In step H, for the X-ray prohibited items dataset, when processing X-ray images, the objectness score and instance score are weighted to obtain the final instance score of the multi-instance branch learning network. This weighting operation is intended to ensure that the network can accurately locate and identify prohibited items. Technically speaking, Multiply by the instance score of the k-th point annotation in the MIL branch, i.e., the k-th bag of MIL; larger weights will be assigned to proposal boxes with higher objectness scores; the calculation of the classification score and subsequent MIL loss remains unchanged, and the process can be expressed as

[0076]

[0077] in, Represents weighted instance scores; Softmax() and FC ins () represent the Softmax function and instance sub-branch respectively.

[0078] For natural image datasets, instance-level annotations are used to supervise the objectivity score. This means that detailed annotations are provided for each instance in the natural image so that the network can accurately learn and understand the target information in the natural image. In this way, the rich objectivity information in the instance-level annotations of the natural image dataset can be effectively utilized and applied to the X-ray prohibited items detection task, thereby improving the performance of the network on X-ray images.

[0079] In step I, firstly, some instances with the highest instance scores are selected from the X-ray prohibited items dataset. These instances are regarded as positive samples with high confidence and used as pseudo labels. Then, these pseudo labels are used to train a fully supervised prohibited items detection model. In this way, the existing information in the X-ray prohibited items dataset and natural images can be fully utilized to further improve the performance of the model in detecting prohibited items. This training strategy can not only enhance the model's ability to recognize prohibited items, but also improve its generalization ability to various prohibited items, making the model more reliable and robust in practical applications.

[0080] By adopting the above technical scheme, the present invention has the following beneficial effects compared with the prior art: the present scheme proposes an intra-domain-inter-domain object learning method for point-supervised X-ray contraband detection. Specifically, in order to solve the problem that the existing method relies heavily on time-consuming and labor-intensive box annotation, the present scheme can perform X-ray contraband detection under point supervision through the intra-domain-inter-domain object module. The invention consists of two key modules: an intra-domain object learning (intra-OL) module and an inter-domain object learning (inter-OL) module. The intra-domain object learning module designs a local focal Gaussian masking block and a global random Gaussian masking block to jointly learn the objectivity in the X-ray image. At the same time, the inter-domain object learning module introduces an adversarial learning block and an objectivity block based on wavelet decomposition, which effectively reduces modal differences and transfers the objectivity knowledge learned from natural images with instance-level annotations to X-ray images. Based on the above content, the present invention greatly alleviates the problem of local dominance caused by severe intra-class changes in X-ray images. Experimental results on four X-ray datasets show that the proposed method achieves excellent performance while significantly reducing the annotation cost, thus improving its practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0082] Figure 1 The figure is a flowchart of the entire implementation of the method according to the embodiment of the present invention.

[0083] Figure 2 This is a diagram of the entire network structure of the detection model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0085] This embodiment provides a method for intra-domain and inter-domain object learning of point-supervised X-ray contraband detection, which includes:

[0086] Step A: Prepare an X-ray prohibited items dataset and a natural image dataset. The former only contains point annotations, while the latter contains instance-level annotations.

[0087] The X-ray prohibited items datasets used in this scheme are 4 commonly used datasets in X-ray prohibited items detection, including OPIXray, HIXray, SIXray, and PIDray. The OPIXray dataset contains 5 categories and a total of 8,885 images. The SIXray dataset contains 6 categories and a total of 1,059,231 images. This scheme uses 8,929 images containing prohibited items for training and testing. The HIXray dataset contains 8 categories and a total of 45,364 images. The PIDray dataset contains 12 categories and a total of 124,486 images, of which this scheme uses 47,677 images containing prohibited items for training and testing. These datasets all include instance-level annotations, but in our experimental settings, points are randomly generated according to Gaussian distribution near the center point of the instance-level annotation as the annotation points used in the training process.

[0088] The natural image dataset used is the commonly used COCO14 dataset. In the COCO14 dataset, there are 82,783 images in the training set, 40,504 images in the validation set, and 40,775 images in the test set. In order to alleviate the imbalance problem between the number of images in the X-ray prohibited items dataset and the natural image dataset, this solution randomly selects the same number of images from the COCO14 training set as in the X-ray dataset.

[0089] The X-ray prohibited items dataset and the natural image dataset in step B and step A are input into the feature extractor to obtain feature maps of the two datasets respectively.

[0090] For the feature extractor, this solution uses a combination of ResNet50 and FPN. The feature extractor is constructed by combining ResNet50 and FPN. ResNet50 is a classic convolutional neural network structure with strong feature extraction capabilities. FPN (Feature Pyramid Network) can extract and fuse features at different scales, allowing the network to better handle objects of different sizes.

[0091] This solution combines ResNet50 with FPN to form a multi-level feature extraction network. ResNet50 is responsible for extracting more detailed and local features at the bottom layer, while FPN performs feature fusion and introduction of contextual information at the high layer. This combination can enable the network to have a better receptive field and semantic information, thereby improving the performance of prohibited items detection.

[0092] By using the feature extractor built with ResNet50 and FPN, we can extract features related to prohibited items from the X-ray prohibited items dataset, and also extract features of general objects from the natural image dataset. In this way, we can obtain feature representations for different datasets, providing richer and more accurate information for subsequent prohibited items detection tasks.

[0093] Step C: Design an in-domain object learning network. The network includes local focal Gaussian masking and global random Gaussian masking to mine in-domain object information.

[0094] Combination Figure 2 As shown in the figure, this scheme designs an in-domain object learning network. The network includes local focus Gaussian masking (LFGM) and global random Gaussian masking (GRGM) to mine the object information in the domain. The purpose of designing an in-domain object learning network is to better mine and utilize the object information in the domain. By combining LFGM and GGRM, the network can improve the robustness, generalization ability and recognition accuracy of the model, thereby achieving more accurate and reliable detection and recognition of prohibited items in complex scenes.

[0095] The LFGM and GRGM blocks of this scheme are designed to better utilize the object information in the domain. The LFGM block guides the model to focus on the non-central area of ​​the prohibited items by generating a local Gaussian mask. This method explicitly forces the model to learn the non-central area of ​​the prohibited items by finding the point with the local maximum activation value and generating a Gaussian mask around the point. The benefit of this is that the model no longer pays too much attention to the area near the labeled point, but is able to more fully understand the shape and structure of the entire prohibited item. With the LFGM block, the model is able to learn and utilize various parts of the prohibited items, improving the accuracy of detection and recognition.

[0096] Specifically, considering that the proposal boxes are generated based on the point annotations around the center of the prohibited items, almost all proposal boxes include the annotated central area of ​​the prohibited items. For example, when the model is trained to detect hammers, the proposal boxes are generated around the annotated points. This approach may cause the model to focus on the handles around the annotation points and ignore other non-central discriminative areas (such as the head). To address this issue, we search for points with local maximum activation values ​​and generate local Gaussian masks to explicitly force the model to learn the non-central areas of the prohibited items. Technically, given an X-ray image I n and its corresponding point annotations First, take the picture I n Input backbone network and FPN to obtain feature maps of different scales in L represents the number of feature maps, H l , W l and C l Represent the feature map F n,l The height, width and number of channels of the feature map F n,l For each channel of , we first l )×(βH 1 ) finds the point with the highest activation in the window (β represents the scaling factor), which is used as the average position for generating the local Gaussian mask. The above process can be expressed as:

[0097]

[0098] in Represents the Cth channel feature map The average position on .

[0099] At the same time, for each F n,l , we also define a covariance matrix ∑ l =αR l ,in is a two-dimensional diagonal matrix, and α represents the scaling factor.

[0100] Finally, we apply a local Gaussian mask to each channel of the feature map,

[0101]

[0102] in represents the feature map after masking, represents a local Gaussian mask.

[0103] On the other hand, the GRGM block enhances the robustness and generalization of the model by introducing global random Gaussian masking. This block forces the model to learn and rely on other recognizable features by randomly masking certain areas of the image. Since the masking is randomly generated, the model needs to learn features outside the masked area to predict the target, thereby enhancing the model's ability to recognize different shapes and locations of prohibited items. Through the GRGM block, the model can better adapt to various interferences and changes, improving performance and generalization in complex environments.

[0104] By applying the LFGM block, the model is forced to learn non-central regions. However, the model may still focus on local regions. Therefore, we further design a GRGM block that randomly applies multiple Gaussian masks on the feature map to prevent the model from over-focusing on local discriminative regions and learn the entire object (including central and non-central regions) globally. This approach is conducive to utilizing the object knowledge of the object's intra-modality. Unlike the LFGM block, the GRGM block randomly generates points in each feature map. Specifically, for each feature map F n,l , randomly generate a set of points in Represents the feature map F n,l The mth randomly generated point (x-coordinate and y-coordinate) on the graph, where M represents the total number of generated points. Therefore, for each point Generate a Gaussian mask

[0105] Finally, this scheme applies these Gaussian masks to the feature map F n,l , and subtract them from the original feature map,

[0106]

[0107] Where F″ n,l Represents the feature map after masking.

[0108] In obtaining F′ n,l and F″ n,l Finally, we concatenate these two feature maps and use a 1×1 convolutional layer to reduce their channel number to obtain feature E n,l This feature is then used as the input feature of the MIL branch.

[0109] Step D: Input the X-ray prohibited items dataset in step B into the in-domain object learning network to obtain the enhanced feature map, and send it to the multi-instance branch learning network to predict the instance score of each instance generated around the annotation point.

[0110] The X-ray prohibited items dataset in step B is input into the in-domain object learning network, and after processing, the enhanced feature maps are obtained. These feature maps are sent to the multi-instance branch learning network, whose task is to predict the instance score in each instance generated around the annotation point.

[0111] E. Design an inter-domain object learning network, which includes a wavelet decoupled adversarial learning network and an object module.

[0112] An inter-domain objectivity learning network is designed to effectively transfer the objectivity information extracted from natural images to X-ray images. The network consists of two key components: a wavelet decomposition based adversarial learning (WDAL) network and an objectness block (OB). First, the wavelet decomposition based adversarial learning network uses wavelet decomposition technology to decouple the feature maps of natural images and X-ray images into two parts: style and content. Then, through adversarial learning, the modal differences caused by style differences are reduced, so that the model can better adapt to the characteristics of X-ray images. Secondly, the objectness block utilizes the transferred objectivity knowledge and incorporates it into the multi-instance learning branch. By calculating the objectivity score of each region of interest (RoI) feature, the module can enhance the probability that the entire prohibited item is contained in the proposal box. These two components work together to enable the model to accurately identify and locate prohibited items in X-ray images, improving the efficiency and accuracy of security detection.

[0113] F. The feature maps of the X-ray prohibited items dataset and the natural image dataset in step B are decoupled by wavelet using a wavelet decoupling adversarial learning network, and adversarial learning is performed on the low-frequency style features.

[0114] In the wavelet decomposition adversarial learning network, it is not ideal to directly transfer the object knowledge extracted from natural images to X-ray images due to the significant modality difference between natural images and X-ray images. To solve this problem, this scheme proposes a WDAL module, which first uses wavelet decomposition to decouple the feature maps of natural images and X-ray images into two parts: style and content, and then reduces the modality difference caused by style differences through adversarial learning. Our WDAL module is inspired by the effectiveness of wavelet decomposition in preserving content information.

[0115] Technically, this scheme uses Haar wavelet for wavelet decomposition. Haar wavelet includes four kernels, namely {LL T , L.H. T , H.L. T , HH T}, where L and H are low-pass and high-pass filters respectively, defined as follows:

[0116]

[0117] Wavelet decomposition transforms the feature map M n,l (From X-ray image I n or natural picturesJ n Extracted from) is decomposed into four parts, namely

[0118] A n,l =M n,l *(LL T )

[0119] H n,l =M n,l *(LH T )

[0120] V n,l =M n,l *(HL T )

[0121] D n,l =M n,l *(HH T )

[0122] Where * represents the convolution operation; A n,l Indicates the low-frequency part; H n,l 、V n,l and D n,l Indicates the high frequency part.

[0123] Usually, the low-frequency part mainly corresponds to the style features, capturing the detailed style information of the image; the high-frequency part corresponds to the content features, focusing on local details and edges. n,l Input into a convolutional layer with learnable parameters to obtain the converted style feature A′ n,l . In order to reduce the modality difference, we perform adversarial learning on the converted style features. We regard the natural image dataset and the X-ray image dataset as the source domain and the target domain respectively. We train a domain classifier on the activation value of each converted style feature to predict the modality of each style feature. This method helps to expand the number of training samples for adversarial learning and reduce the global image difference. Mathematically, let is the domain label of the nth image, where if the nth image belongs to the source domain, then Otherwise, it is 1. This scheme expresses the activation value of the converted style feature at the coordinate (x, y) as Using corresponding style features The predicted value for each position Calculate the cross-entropy based adversarial loss L adv ,Right now

[0124]

[0125] To align the modality distributions, we simultaneously optimize the parameters of the modality classifier, by minimizing the adversarial loss, and optimize the parameters of the base network, by maximizing this loss. We also use gradient reversal layers (GRL) to train the modality classifier.

[0126] Then, the content feature Hn,l 、V n,l and D n,l and the converted style feature A′ n,l Connect and get the feature H without modality specificity n,l ,Right now

[0127]

[0128] in Represents a concatenation operation.

[0129] G. Use the objectness module to predict the objectness score of each target in the X-ray security image dataset and the natural image dataset

[0130] Using the objectness module, we predict the objectness score of each object in the X-ray security image dataset and the natural image dataset. After using the WDAL module to transfer the objectness information extracted from natural images to X-ray images, we designed an objectness module to incorporate this information into the MIL branch, enhancing the probability that the proposal box contains the entire prohibited item.

[0131] This scheme computes an objectness score that reflects the probability that each region of interest (RoI) feature contains the entire object. represents the proposed RoI feature generated by the kth annotation point of the nth image (X-ray or natural image). Here, R represents the total number of proposals. We then perform RoI pooling on each feature map and calculate the final objectness score through a fully connected layer. The objectness module can be expressed as:

[0132]

[0133] Where RoI_Pooling() and FC() represent RoI pooling and fully connected layers respectively; Represents the objectness score corresponding to the r-th proposal box generated by the k-th point.

[0134] For natural images, we can leverage the rich box annotations to train the objectness module. Specifically, for the proposal boxes generated from the point annotations, we compute the intersection-over-union ratio between them and the annotated box annotations. Among them, we regard the proposal box with an intersection-over-union ratio greater than or equal to 0.5 as a positive sample, marked as 1, and the proposal box with an intersection-over-union ratio less than this value as a negative sample, marked as 0. Based on this, we can use the cross entropy loss function to adjust the model parameters, that is,

[0135]

[0136] here, and They represent the areas annotated by the rth proposal box and its corresponding annotation box, respectively.

[0137] Finally, I 2 The joint loss of OL-Net is

[0138] L=L mil +λ 1 L adv +λ 2 L obj

[0139] Where L mil represents the total MIL loss defined in P2BNet; λ 1 and λ 2 is the balance weight.

[0140] H. Weight the objectness score of the X-ray prohibited items dataset and the instance score in D to obtain the final instance score of the multi-instance branch learning network. For the natural image dataset, its objectness score is supervised by instance-level annotation.

[0141] For the X-ray prohibited items dataset, when processing X-ray images, this scheme weights the objectness score and instance score to obtain the final instance score of the multi-instance branch learning network. This weighting operation is intended to ensure that the network can accurately locate and identify prohibited items. Technically speaking, Multiply by the instance score of the k-th point annotation in the MIL branch, i.e., the k-th bag of MIL; larger weights will be assigned to proposal boxes with higher objectness scores; the calculation of the classification score and subsequent MIL loss remains unchanged, and the process can be expressed as

[0142]

[0143] in, Represents weighted instance score; Soffmax() and FC ins () represent the Softmax function and instance sub-branch respectively.

[0144] For the natural image dataset, we use instance-level annotations to supervise the objectness score. This means that we provide detailed annotations for each instance in the natural image so that the network can accurately learn and understand the target information in the natural image. In this way, we can effectively utilize the rich objectness information in the instance-level annotations of the natural image dataset and apply it to the X-ray prohibited items detection task, thereby improving the network's performance on X-ray images.

[0145] I. Select the instances with the highest instance scores from the X-ray prohibited items dataset as pseudo labels to train a fully supervised prohibited items detection model.

[0146] This scheme first selects some instances with the highest instance scores from the X-ray prohibited items dataset. These instances are regarded as positive samples with high confidence and are used as pseudo labels.

[0147] We then used these pseudo labels to train a fully supervised prohibited items detection model. This approach allows us to fully leverage existing information in the X-ray prohibited items dataset and in natural images to further improve the model’s performance in detecting prohibited items. This training strategy not only enhances the model’s ability to identify prohibited items, but also improves its generalization to a variety of prohibited items, making the model more reliable and robust in practical applications.

[0148] In order to facilitate the comparative test of the performance of the proposed method, the proposed method is evaluated on four commonly used X-ray datasets, including OPIXray, HIXray, SIXray and PIDray. Among them, the OPIXray dataset contains 5 categories and a total of 8,885 images. The SIXray dataset contains 6 categories and a total of 1,059,231 images. We use 8,929 images containing prohibited items for training and testing. The HIXray dataset contains 8 categories and a total of 45,364 images. The PIDray dataset contains 12 categories and a total of 124,486 images. This scheme uses 47,677 images containing prohibited items for training and testing.

[0149] During the training process, this scheme also uses the COCO14 training set as an additional dataset. In order to balance the imbalance in the number of images between the natural image dataset and the X-ray dataset, this scheme randomly selects the same number of images from the COCO14 training set as the X-ray dataset, and merges the randomly selected subset with the X-ray training set to construct the final training set.

[0150] This method is built on P2BNet and implemented using MMDetection. This method uses 2 NVIDIA RTX 3090 GPUs to train the model, using the SGD algorithm with a learning rate of 0.005. The total number of training rounds is set to 12 and the batch size is set to 4. The data of each batch is randomly selected from the X-ray training set and the additional dataset of natural images to fully utilize the parallelism of the GPU. In the loss function, the balancing weights λ_1 and λ_2 are set to 0.1 and 1, respectively. The scaling factors α and β are set to 0.1 and 0.15, respectively. The number of random points M in GRGM is set to 2. Except for the parameters mentioned above, all other settings remain the same as P2BNet.

[0151] Correspondingly, the model used as the control network is trained using a training method or parameters similar to that of this scheme; the performance comparison of this scheme with other weakly supervised target detectors, point-supervised target detectors, and fully supervised target detectors on the OPIXray+HIXray+SIXray+PIDray dataset is shown in Table 1 below:

[0152] Table 1

[0153]

[0154] Note: Faster R-CNN corresponds to the method proposed by Ren, S. et al. (Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems 28 (2015));

[0155] RetinaNet corresponds to the method proposed by Lin, TY, et al. (Lin, TY, Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE / CVF International Conference on Computer Vision. pp. 2980–2988 (2017));

[0156] YOLOv8-s corresponds to the method proposed by Jocher, G. et al. (Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics YOLO (Jan2023), https: / / github.com / ultralytics / ultralytics);

[0157] FCOS corresponds to the method proposed by Tian, ​​Z. et al. (Tian, ​​Z., Shen, C., Chen, H., He, T.: FCOS: Fully convolutional one-stage object detection. In: Proceedings of the IEEE / CVF International Conference on Computer Vision. pp. 9627–9636 (2019));

[0158] Deformable DETR corresponds to the method proposed by Zhu, X. et al. (Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159(2020));

[0159] PCL corresponds to the method proposed by Tang, P. et al. (Tang, P., Wang, X., Bai, S., Shen, W., Bai, X., Liu, W., Yuille, A.: PCL: Proposal cluster learning for weakly supervised object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(1), 176–191(2018));

[0160] WSOD2 corresponds to the method proposed by Zeng, Z. et al. (Zeng, Z., Liu, B., Fu, J., Chao, H., Zhang, L.: WSOD2: Learning bottom-up and top-down objectness distillation for weakly-supervised object detection. In: Proceedings of the IEEE / CVF International Conference on Computer Vision. pp. 8292–8300 (2019));

[0161] OD-WSCL corresponds to the method proposed by Seo, J. et al. (Seo, J., Bae, W., Sutherland, DJ, Noh, J., Kim, D.: Object discovery via contrastive learning for weakly supervised object detection. In: Proceedings of the European Conference on Computer Vision. pp. 312–329 (2022));

[0162] P2BNet corresponds to the method proposed by Chen, P. et al. (Chen, P., Yu, X., Han, X., Hassan, N., Wang, K., Li, J., Zhao, J., Shi, H., Han, Z., Ye, Q.: Point-to-box network for accurate object detection via single point supervision. In: Proceedings of the European Conference on Computer Vision. pp. 51–67 (2022)).

[0163] As can be seen from the above comparison results, the fully supervised prohibited items detection model obtained by this method and its corresponding training has obvious progress and advantages compared with the solutions disclosed in some existing literature; this solution can effectively mine the object information in the domain, and can transfer the rich object information contained in the instance-level annotation of the natural image dataset to the X-ray prohibited items dataset, which greatly alleviates the local dominance problem caused by the serious intra-domain changes in the X-ray images. The invention has achieved good performance in multiple X-ray prohibited items datasets, and the performance has been greatly improved compared with the traditional weakly supervised target detection algorithm and the point-supervised target detection algorithm.

[0164] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for intra-domain and inter-domain object learning for point-supervised X-ray contraband detection, characterized in that: It includes: A. Prepare a dataset of X-ray prohibited items and a dataset of natural images, where the former only contains annotation points and the latter contains instance-level annotations; B. Input the X-ray prohibited items dataset and the natural image dataset into the feature extractor to obtain the feature maps corresponding to the two datasets respectively; C. Design an in-domain object learning network, which includes local focal Gaussian masking and global random Gaussian masking to mine in-domain object information; D. Input the feature map of the X-ray prohibited items dataset obtained in step B into the in-domain object learning network to obtain an enhanced feature map, and then send it to the multi-instance branch learning network to predict the instance score of each instance generated around its annotation point; E. Design an inter-domain object learning network, which includes a wavelet decoupled adversarial learning network and an object module; F. Perform wavelet decoupling on the feature graph of the X-ray prohibited items dataset and the feature graph of the natural image dataset obtained in step B using a wavelet decoupling adversarial learning network, and perform adversarial learning on the low-frequency style features; G. Use the objectness module to predict the objectness score of each target in the feature map of the X-ray prohibited items dataset and the natural image dataset; H. Weighting the objectness score of the feature graph of the X-ray prohibited items dataset and the instance score obtained by processing in step D to obtain the final instance score of the multi-instance branch learning network; For natural image datasets, their objectness scores are supervised by instance-level annotations; I. Select instances whose instance scores of the X-ray prohibited items dataset are higher than the preset value, use them as pseudo labels, and then train a fully supervised prohibited items detection model.

2. The intra-domain and inter-domain object learning method for point-supervised X-ray contraband detection according to claim 1, characterized in that: In step A, the X-ray prohibited items dataset includes OPIXray dataset, HIXray dataset, SIXray dataset, and PIDray dataset; the OPIXray dataset contains 5 categories, totaling 8,885 images; the SIXray dataset contains 6 categories, totaling 1,059,231 images; the HIXray dataset contains 8 categories, totaling 45,364 images; the PIDray dataset contains 12 categories, totaling 124,486 images; Among them, the OPIXray dataset, HIXray dataset, SIXray dataset, and PIDray dataset all include instance-level annotations, and points are randomly generated according to Gaussian distribution near the center point of the instance-level annotation as the annotation points used in the training process; The natural image dataset is the commonly used COCO14 dataset; the same number of images as the X-ray dataset are randomly selected from the COCO14 training set to alleviate the imbalance problem in the number of images in the X-ray prohibited items dataset and the natural image dataset.

3. The intra-domain and inter-domain object learning method for point-supervised X-ray contraband detection according to claim 1 or 2, characterized in that: In step B, the feature extractor is constructed by combining ResNet50 and FPN. ResNet50 and FPN are combined to form a multi-level feature extraction network. ResNet50 is responsible for extracting more detailed and local features at the bottom layer, and FPN is responsible for feature fusion and introduction of context information at the high layer. By combining the feature extractor built with ResNet50 and FPN, features related to prohibited items are extracted from the X-ray prohibited items dataset. At the same time, features of general objects are extracted from the natural image dataset. Feature maps corresponding to the two datasets are obtained, which are set as natural images and X-ray images respectively, to obtain feature representations for different datasets.

4. The intra-domain and inter-domain object learning method for point-supervised X-ray contraband detection as claimed in claim 3, characterized in that: Step C includes: Design an in-domain object learning network, which includes a local focal Gaussian mask (LFGM) and a global random Gaussian mask (GRGM) to mine in-domain object information; Among them, Local Focus Gaussian Masking LFGML guides the model to focus on the non-central area of ​​prohibited items by generating a local Gaussian mask; by finding the point with the local maximum activation value and generating a Gaussian mask around the point, the model is explicitly forced to learn the non-central area of ​​prohibited items; its principles include: For a given X-ray image I n and its corresponding point annotations First, take the picture I n Input backbone network and FPN to obtain feature maps of different scales in L represents the number of feature maps, H l , W l and C l Represent the feature map F n,l The height, width and number of channels of the feature map F n,l Each channel of l )×(βH l ) finds the point with the highest activation in the window, β represents the scaling factor, and this point is used as the average position for generating the local Gaussian mask; the above process can be expressed as: in, Represents the c-th channel feature map The average position on At the same time, for each F n,l Define a covariance matrix Σ l =αR l ,in is a two-dimensional diagonal matrix, α represents the scaling factor; Finally, a local Gaussian mask is applied to each channel of the feature map, in, represents the feature map after masking, represents a local Gaussian mask; Global Random Gaussian Masking GRGM enhances the robustness and generalization ability of the model by introducing global random Gaussian masking, which randomly applies multiple Gaussian masks on the feature map to prevent the model from over-focusing on local discriminative areas and globally learn the entire object, including the center and non-center areas; its principles include: For each feature map F n,l , randomly generate a set of points in Represents the feature map F n,l The mth randomly generated point (x coordinate and y coordinate) on the graph, where M represents the total number of points generated; based on this, for each point Generate a Gaussian mask Apply these Gaussian masks to the feature map F n,l , and subtract them from the original feature map, Where F n,l Represents the feature map after masking; In obtaining F′ n,l and F″ n,l Finally, the two feature maps are concatenated and a 1×1 convolutional layer is used to reduce their channel number to obtain feature E n,l , which is used as the input feature of the MIL branch.

5. The intra-domain and inter-domain object learning method for point-supervised X-ray contraband detection according to claim 4, characterized in that: Step E includes: Design an inter-domain objectivity learning network to effectively transfer objectivity information extracted from natural images to X-ray images; the network consists of two key components, including a wavelet decoupled adversarial learning network and an objectivity module; Among them, the wavelet decoupling adversarial learning network uses wavelet decomposition technology to decouple the feature graphs of the X-ray prohibited items dataset and the natural image dataset, dividing them into two parts: style and content; then, through adversarial learning, the modal differences caused by style differences are reduced, so that the model can better adapt to the characteristics of X-ray images; The objectness module utilizes the transferred objectness knowledge and integrates it into the multi-instance learning branch. By calculating the objectness score of each region of interest (RoI) feature, the module is able to enhance the probability that the proposal box contains the entire prohibited item.

6. The intra-domain and inter-domain object learning method for point-supervised X-ray contraband detection according to claim 5, characterized in that: Step F comprises: In the wavelet decomposition adversarial learning network, the feature maps of the X-ray prohibited items dataset and the feature maps of the natural image dataset are decoupled into two parts, style and content, using wavelet decomposition through the WDAL module. Then, adversarial learning is used to reduce the modal differences caused by style differences. Use Haar wavelet for wavelet decomposition. Haar wavelet includes four kernels, namely {LL T ,LH T ,HL T ,HH T }, where L and H are low-pass and high-pass filters respectively, defined as follows: Wavelet decomposition transforms the feature map M n,l , which is from the X-ray image I n or natural picturesJ n Extract it and decompose it into four parts, namely A n,l =M n,l *(LL T ) H n,l =M n,l *(LH T ) <h2 style=";text-align:left;direction:ltr">V<h2 style=";text-align:left;direction:ltr"> n,l <h2 style=";text-align:left;direction:ltr"> =M<h2 style=";text-align:left;direction:ltr"> n,l <h2 style=";text-align:left;direction:ltr"> *(HL<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> ) D n,l =M n,l *(HH T ) Where * represents the convolution operation; A n,l Indicates the low-frequency part; H n,l 、V n,l and D n,l Represents the high frequency part; Among them, the low-frequency part mainly corresponds to the style features, capturing the detailed style information of the image; the high-frequency part corresponds to the content features, focusing on local details and edges; then, A n,l Input into a convolutional layer with learnable parameters to obtain the converted style feature A′ n,l ; In order to reduce the modality difference, adversarial learning is performed on the converted style features, and the natural image dataset and the X-ray prohibited items dataset are regarded as the source domain and the target domain respectively; a domain classifier is trained on the activation value of each converted style feature to predict the modality of each style feature; in mathematical principle, let is the domain label of the nth image, where if the nth image belongs to the source domain, then Otherwise, it is 1; the activation value of the converted style feature at the coordinate (x, y) is expressed as Using corresponding style features The predicted value for each position Calculate the cross-entropy based adversarial loss L adv ,Right now To align the modal distributions, we simultaneously optimize the parameters of the modal classifier, by minimizing the adversarial loss, and the parameters of the base network, by maximizing this loss. We also use a gradient reversal layer GRL to train the modal classifier. Then, the content feature H n,l 、V n,l and D n,l And the converted style feature A′ n,l Connect and get the feature H without modality specificity n,l ,Right now in Represents a concatenation operation.

7. The intra-domain and inter-domain object learning method for point-supervised X-ray contraband detection according to claim 6, characterized in that: Step G includes: The objectivity module is used to predict the objectivity score of each target in the feature map of the X-ray prohibited items dataset and the natural image dataset. The objectivity information extracted from the natural image is transferred to the X-ray image using the WDAL module. This information is then integrated into the MIL branch of the objectivity module to enhance the probability that the proposal box contains the entire prohibited item. The objectness score reflects the probability that each region of interest RoI feature contains the entire object; it includes: represents the proposed RoI feature generated by the kth annotation point of the nth X-ray prohibited items image or natural image, where R represents the total number of proposals; RoI pooling is performed on each feature map, and the final objectness score is calculated through a fully connected layer. The objectness module can be expressed as: Among them, RoI_Pooling() and FC() represent RoI pooling and fully connected layers respectively; Represents the objectness score corresponding to the r-th proposal box generated by the k-th point; For natural images, rich box annotations can be used to train the objectness module. Specifically, for the proposal boxes generated from the point annotations, the intersection-over-union ratio between them and the annotated boxes is calculated. Among them, the proposal box with an intersection-over-union ratio greater than or equal to 0.5 is regarded as a positive sample and marked as 1, and the proposal box with an intersection-over-union ratio less than this value is regarded as a negative sample and marked as 0. Based on this, the cross entropy loss function can be used to adjust the model parameters, that is, in, and Respectively represent the area of ​​the r-th proposal box and its corresponding annotation box; Finally, I 2 The joint loss of OL-Net is L=L mil +λ1L adv +λ2L obj Among them, L mil represents the total MIL loss defined in P2BNet; λ1 and λ2 are balancing weights.

8. The intra-domain and inter-domain objectivity learning method for point-supervised X-ray contraband detection according to claim 7, characterized in that: In step H, the objectness score of the feature graph of the X-ray prohibited items dataset and the instance score obtained by processing in step D are weighted to obtain the final instance score of the multi-instance branch learning network including: For X-ray images, Multiply by the instance score of the k-th point annotation in the MIL branch, i.e., the k-th bag of MIL; larger weights will be assigned to proposal boxes with higher objectness scores; the calculation of the classification score and subsequent MIL loss remains unchanged, and the process can be expressed as: in, Represents weighted instance scores; Softmax() and FC ins () represent the Softmax function and instance sub-branch respectively.

Citation Information

Patent Citations

  • Forbidden object detection method based on multi-scale cross-image weak supervised learning

    CN113159120A

  • Method and system for detecting objects in images

    EP4120132A1

  • Learning apparatus, learning method, object detection apparatus, object detection method, learning support system and learning support method

    US20230132770A1