Contraband classification method for X-ray security inspection images based on multi-view and multi-label information bottleneck

Abstract

The present invention discloses a multi-view multi-label X-ray security inspection image contraband classification method based on information bottleneck. First, a hierarchical feature extractor is used to extract features; a multimodal feature representation is generated based on the extracted features; then, based on the obtained multimodal feature representation, a prediction value for each view is generated respectively; finally, a comprehensive prediction is performed based on the prediction values ​​of multiple views. The present invention uses a multi-view multi-label information bottleneck algorithm composed of a label-aware information bottleneck module and an adaptive lightweight semantic image interaction module to maximize the shared information between different views while minimizing the influence of noise information within the view under the condition of label independence. In addition, the present invention utilizes a consistency-specific information processing algorithm and redesigns the information calculation correlation function to maximize the shared information while minimizing the correlation between the specific information in a single view and other views, as well as the difference between the specific information and the shared information.

Description

Technical Field

[0001] The present invention belongs to the field of multi-view multi-label classification learning and X-ray security inspection, and specifically relates to a multi-view multi-label X-ray security inspection image contraband classification method based on information bottleneck. Background Art

[0002] With the continuous increase in the number of people at public transportation stations, ensuring public safety has become particularly critical. X-ray security inspection technology has become an important means to ensure the safety of public places. After luggage, backpacks and other items are exposed to X-rays, some of the items inside will be revealed, helping security personnel to detect potential dangerous goods and ensure the safety of public places. Due to the penetrating nature of X-rays, compared with natural light images, X-ray images have the following characteristics: (1) Overlapping: When stacked together, objects will overlap in the image under X-ray irradiation, resulting in blurred surface textures of the objects in the image; (2) Hidden: Contraband in X-ray security images is small in size and overlaps with other items, making it difficult for the system to detect them under a single viewing angle; (3) Difficulty in labeling: Due to characteristics (1) and (2), object-level labeling of X-ray security images requires a huge amount of time and economic cost.

[0003] Identifying prohibited items in X-ray security images can be considered a multi-label classification task. Compared to object detection, object classification does not require object-level annotation of X-ray security datasets, requiring simpler and more efficient image-level annotation. This means that object classification can be used to determine whether a suitcase contains prohibited items based on overall image features. In addition to prohibited items, suitcases often contain a variety of common items, known as accompanying items. While these accompanying items are generally irrelevant to the classification task, their features can interfere with the identification of prohibited items during the classification process. Therefore, filtering and suppressing these interfering features is necessary during the classification process. Furthermore, because stacked prohibited items overlap in X-ray security images, a single layer of visual features may not fully capture the key information about these overlapping prohibited items. Furthermore, when detecting prohibited items in X-ray security images from a single viewing angle, some prohibited items may be difficult to identify due to limited viewing angles. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention provides a multi-view and multi-label X-ray security inspection image contraband classification method based on information bottleneck.

[0005] The multi-view multi-label X-ray security inspection image contraband classification method based on information bottleneck includes the following steps:

[0006] The top view and side view X-ray security inspection images are obtained as the input of the classification model, and the corresponding hierarchical feature extractors are used for feature extraction.

[0007] A multimodal feature representation is generated based on the extracted features, and a prediction value for each view is generated separately; a comprehensive prediction is performed based on the prediction values ​​of multiple views.

[0008] Set the total loss function and iteratively train the classification model; verify it on the validation set to obtain the optimal parameter model and output the contraband classification results.

[0009] In a possible implementation, a dual-view image set including X-ray security inspection images from a top view perspective and a side view perspective is divided into a training set and a validation set;

[0010] Resize the image and normalize the pixel values;

[0011] For the input image, the top view perspective level feature extractor and the side view perspective level feature extractor are used to generate features under the shared view and the individual view, namely the image feature representation.

[0012] In one possible implementation, the NLP language model BERT is used to generate an embedded representation of the label space; based on the extracted features and the embedded representation of the label space, the adaptive lightweight semantic image interaction module ALSI is used to generate a multimodal feature representation. Specifically, first, the image feature representation After the image feature mapping matrix Mapping, at the same time, embedding the label space into representation Using label embedding mapping matrix Mapping is performed, the two mapped features are Hadamard-multiplied, and a nonlinear activation function is applied to the product result; finally, the adaptive weight matrix Perform weighted processing to obtain multimodal feature representation.

[0013] In one possible implementation, the multimodal feature representation is mapped to prediction values ​​for all views of each input image sample through MLP mapping.

[0014] In one possible implementation, according to The predicted values ​​of all views of the k-th input image sample are used to make a comprehensive prediction, and finally the multi-view prediction output value is obtained. Calculate, then perform Hadamard product calculation on the calculation result and all views of the original k input image samples, and sum the calculation results to obtain the final prediction value.

[0015] In one possible implementation, the total loss function includes a common feature information calculation loss function, a correlation loss function between shared views and single view specific features, a correlation loss function between each different single view, and a prediction loss function.

[0016] According to Inference 1 and Inference 2 in the label-aware information bottleneck module LAIB, the loss function is calculated by calculating the common feature information.

[0017] In the consistency-specific information processing module CSIP, the correlation loss function between the specific features of the shared view and the single view, as well as the correlation loss function between each different single view are calculated according to the consensus principle and the complementary principle.

[0018] The prediction loss function uses binary cross entropy loss.

[0019] In one possible implementation, a prepared training set is input into a classification model, and multiple rounds of training are performed until convergence. A total loss function is used to measure the difference between the model's predicted output and the true label, and the loss function is minimized to improve the model's prediction accuracy. During the training process, the mAP indicator is used as a measurement metric to continuously monitor the model's performance on the validation set. Whenever the best mAP value is obtained during training, the corresponding model configuration is recorded and saved as the optimal parameters.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The multi-view multi-label X-ray security inspection image contraband classification method proposed in this invention based on information bottleneck improves the classification performance of X-ray security inspection image contraband. Specifically, the present invention uses a multi-view multi-label information bottleneck MMIB algorithm composed of a label-aware information bottleneck module LAIB and an adaptive lightweight semantic image interaction module ALSI. Under the condition of label independence, the method maximizes the shared information between different views while minimizing the influence of noise information within the view. Among them, the LAIB module is in the latent space defined by the labels and image samples. In the application, the information bottleneck technique and its label-related latent variables , while maximizing task-related shared information and minimizing internal redundancy in the view. The ALSI module dynamically adjusts the weight information between image features and label embedding representations, improving the model's sensitivity to key image features during multimodal feature learning. In addition, the present invention utilizes the Consistency Specificity Information Processing (CSIP) algorithm, through multi-view consensus and complementary principles, to maximize shared information while minimizing the correlation between specific information in a single view and other views, as well as the difference between this specific information and the shared information, thereby improving the algorithm's classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flowchart of a method according to an embodiment of the present invention.

[0023] Figure 2 Schematic diagram of the overall structure of the model of an embodiment of the present invention.

[0024] Figure 3 Schematic diagram of the hierarchical feature extractor structure of an embodiment of the present invention.

[0025] Figure 4 Schematic diagram of the ALSI module structure according to an embodiment of the present invention.

[0026] Figure 5 Comparison results of classification mAP. DETAILED DESCRIPTION

[0027] In order to better understand the purpose, structure and function of the present invention, the technical solution of the present invention is further described in detail below with reference to the accompanying drawings. The present invention proposes a multi-view multi-label X-ray security inspection image contraband classification method based on information bottleneck. The overall process is as follows Figure 1 The overall structure of the model is shown in Figure 2 As shown in the figure, the present invention uses a multi-view multi-label information bottleneck MMIB algorithm composed of a label-aware information bottleneck module LAIB and an adaptive lightweight semantic image interaction module ALSI to maximize the shared information between different views under the condition of label independence and minimize the influence of noise information within the view. In the application, the information bottleneck technique and its label-related latent variables , aims to maximize task-related shared information and minimize intra-view redundancy. ALSI dynamically adjusts the weight information between image features and label embedding representations to improve the model's sensitivity to key image features in multimodal feature learning, such as Figure 4 In addition, the present invention utilizes the Consistency Specificity Information Processing (CSIP) algorithm to maximize shared information through multi-view consensus and complementary principles, while minimizing the correlation between specific information in a single view and other views, as well as the difference between this specific information and the shared information, thereby improving the classification performance of the algorithm.

[0028] In one possible implementation, a hierarchical feature extractor is used to extract features. The specific operations are as follows:

[0029] (1-1) The acquired dual-view image set is divided into a training set and a validation set: the training set is used for model training, i.e. ; The validation set is used to verify the accuracy of the model, that is . Where R is the real number field, represents the number of image samples in the training set, represents the i-th training image sample, represents the number of image samples in the validation set, Represents the j-th verification image sample, H represents the image height, W represents the image width, and 3 represents the number of RGB channels.

[0030] (1-2) Each training sample Corresponding tags ;in Represents an image sample The number of targets contained in express Middle The true categories of the targets are represented by C, where C represents the total number of categories in the dataset. The images are resized to 224×224 pixels and pixel values ​​are normalized to ensure the consistency and validity of the model input.

[0031] (1-3) For the input image, through the hierarchical feature extractor (Specifically including top view perspective level feature extractor and side view view level feature extractor ), generate features under shared view and separate view, namely image feature representation: , , as well as , where the superscript Indicates the public feature information under the shared view, with superscript Represents specific feature information within a single view.

[0032] Among them, in the hierarchical feature extractor In the

[15] , after extracting features from each feature extraction layer, the extracted features are globally averaged pooled and then feature concatenation is performed to obtain image feature representation.

[0033] In one possible implementation, a multimodal feature representation is generated based on the extracted features. The specific operations are as follows:

[0034] (2-1) Use the pre-trained BERT language model to generate the embedding representation of each contraband label ,in Indicates the dimension of each label embedding representation.

[0035] (2-2) Representation by image features 、 and label embedding representation As the input of the adaptive lightweight semantic image interaction module ALSI, it generates multimodal feature representation .

[0036] In one possible implementation, based on obtaining a multimodal feature representation , generate the predicted value of each view respectively. That is, use MLP mapping to The first input image sample Views are mapped to predicted values .

[0037] In one possible implementation, a comprehensive prediction is performed based on the prediction values ​​of multiple views. Calculate, then perform Hadamard product calculation on the calculation result and all views of the original k input image samples, and sum the calculation results to obtain the final prediction value.

[0038] In one possible implementation, the loss function is calculated based on the first and second inferences in the label-aware information bottleneck module LAIB. Specifically, Corollary 1:

[0039]

[0040] in, Represents the joint input of image X and label Y. is the mutual information function. Represents entropy. is the prior probability A variational approximation of . In practical applications, It is usually assumed to be a spherical Gaussian distribution. It uses the empirical data distribution ,in is the Dirac function, which is used to represent the distribution of data points. is the total number of data points, It is data points.

[0041] Corollary 2:

[0042]

[0043] According to Corollary 1 and Corollary 2, the loss function is calculated based on the public feature information. The definition is as follows:

[0044]

[0045] Where E represents the expected operation.

[0046] In a possible implementation, in the consistency specific information processing module CSIP, the correlation loss function between the specific features of the shared view and the single view is calculated according to the consensus principle and the complementarity principle. , and the correlation loss function between each different single view , step (6) is specifically performed as follows:

[0047] (6-1) In the consistency specificity information processing module CSIP It is a loss function that measures the correlation between shared information between views and specific information within a single view. With specific characteristics , = ], = ],v , ]; its correlation loss function By calculation After multiplying the products and taking the logarithm, the sum is obtained.

[0048] (6-2): CSIP It is a loss function that measures the correlation between the specific information in each view and the specific information in other views. .

[0049] By utilizing the consistency-specific information processing module (CSIP), under the principle of multi-view consensus and complementarity, the algorithm maximizes shared information while minimizing the correlation between specific information in a single view and other views, as well as the difference between this specific information and the shared information, thereby improving the classification performance of the algorithm.

[0050] In one possible implementation, the model is iteratively trained based on a designed total loss function, and validated on a validation set to obtain an optimal parameter model and output a contraband classification result. The specific operations are as follows:

[0051] (7-1) The total loss function is designed as follows:

[0052]

[0053] in, is the function calculation parameter, that is yes The calculation parameters of yes The calculation parameters of yes The calculation parameters of yes Calculation parameters. is the prediction loss function, which is calculated using binary cross entropy loss.

[0054] (7-2): Input the prepared training set into the model framework (prediction model) constructed in steps 1-4 and perform multiple rounds of training until convergence. A loss function is used to measure the difference between the model's predicted output and the true label, and the loss function is minimized to improve the model's prediction accuracy. During the training process, the mAP metric is used as a measurement to continuously monitor the model's performance on the validation set. Whenever the best mAP value is obtained during training, the corresponding model configuration is recorded and saved as the optimal parameters. This approach helps find the optimal parameter combination during training, thereby improving the model's prediction accuracy.

[0055] (7-3): After the training is completed, the validation set is used to verify the optimal model obtained in step (7-2), and the indicator parameters of the final model for detecting the data set are obtained. The category of the detected contraband and the confidence level are marked on the test results.

[0056] Experimental comparison data:

[0057] The dataset uses DvXray and OPIXray.

[0058] (1) DvXray is the first dual-view X-ray security inspection image dataset. It contains 15 prohibited items: gun, knife, wrench, pliers, scissors, lighter, battery, bat, razor blade, saw blade, fireworks, hammer, screwdriver, dart, and pressure vessel. The following prohibited items are abbreviated as: Wrench (Wren), Scissors (Scissors), Battery (Batt), Razor blade (Razor), Saw blade (Saw), Fireworks (Fires), Hammer (Hamm), Screwdriver (Screw), Pressure vessel (PV). The DvXray dataset is shown in Table 1.

[0059] Table 1: DvXray dataset

[0060]

[0061] (2) The OPIXray dataset contains 8,885 images of prohibited items in five knife categories: folding knives, straight knives, scissors, utility knives, and multi-tool knives. OPIXray focuses on occluded prohibited knives, which are usually small and easily obscured by other objects, posing new challenges to X-ray security inspections. The OPIXray dataset is shown in Table 2.

[0062] Table 2: OPIXray dataset

[0063]

[0064] Experimental comparison description:

[0065] Figure 5 The paper shows the mAP (cmAP) comparison results of the proposed multi-view multi-label X-ray security image contraband classification method X-MP based on information bottleneck and other algorithms on the DvXray dataset. These algorithms include: CHR, DOAM, SXMNet, LIM, AHCR, MVCNN-new, MHBN, and GVCNN. Figure 5 As can be seen from the figure, X-MP achieves the best result of 85.6%. This is 12.5% ​​higher than AHCR, 16.3% higher than CHR, and 21% higher than GVCNN. This demonstrates that X-MP has excellent performance in identifying contraband.

[0066] Table 3: Comparison of X-MP detection results on OPIXray

[0067]

[0068] Table 3 shows the comparison results between X-MP and AHCR on the OPIXray dataset. As shown in Table 1, when X-MP uses ResNet-50 as the backbone network for feature extraction, i.e., the hierarchical feature extractor, its cmAP is 8.33% higher than that of AHCR. X-MP using ResNet-50 achieves a 5.42% higher cmAP than Swin-T using it as the hierarchical feature extractor. Furthermore, ResNet-50 outperforms Swin-T by 8.36% in identifying folding knives, suggesting that limitations of the self-attention mechanism may cause Swin-T to miss key features when detecting small and overlapping prohibited items, resulting in decreased performance.

[0069] The above embodiments have been described in detail to illustrate the objectives, technical solutions, and advantages of the present invention. It should be understood that these embodiments represent only some preferred implementations of the present invention and are not intended to limit its scope. Any modification, equivalent replacement, or improvement, subject to the core concepts and principles of the present invention, should be considered to be within the scope of protection of the present invention.

Claims

1. A multi-view and multi-label X-ray security inspection image contraband classification method based on information bottleneck, characterized by: The following steps are involved: Obtain top-view and side-view X-ray security inspection images as input to the classification model, and use the corresponding hierarchical feature extractors for feature extraction; Generate multimodal feature representation based on the extracted features, generate prediction values ​​for each view separately; and make comprehensive predictions based on the prediction values ​​of multiple views; Using the NLP language model BERT, we generate an embedded representation of the label space. Based on the extracted features and the embedded representation of the label space, we use the adaptive lightweight semantic image interaction module to generate a multimodal feature representation. Specifically, we first use the image feature representation. After the image feature mapping matrix Mapping, at the same time, embedding the label space into representation Using label embedding mapping matrix Mapping is performed, the two mapped features are Hadamard-multiplied, and a nonlinear activation function is applied to the product result; finally, the adaptive weight matrix Perform weighted processing to obtain multimodal feature representation; Set the total loss function and iteratively train the classification model; verify it on the validation set to obtain the optimal parameter model and output the contraband classification results; The total loss function includes the loss function for calculating common feature information, the correlation loss function between the specific features of the shared view and the single view, the correlation loss function between each different single view, and the prediction loss function; Public feature information calculation loss function The definition is as follows: Among them, E represents the expected operation; According to the consensus principle and the complementary principle, the correlation loss function between the specific features of the shared view and the single view, as well as the correlation loss function between each different single view are calculated.

2. The multi-view multi-label X-ray security inspection image contraband classification method based on information bottleneck according to claim 1 is characterized by: A dual-view image set including top view and side view X-ray security inspection images is divided into a training set and a validation set; Resize the image and normalize the pixel values; For the input image, the top view perspective level feature extractor and the side view perspective level feature extractor are used to generate features under the shared view and the individual view, namely the image feature representation.

3. The multi-view multi-label X-ray security inspection image contraband classification method based on information bottleneck according to claim 2 is characterized by: An MLP mapping is built, which takes the obtained multimodal feature representation as input and generates predictions for all views of each input image sample.

4. The multi-view multi-label X-ray security inspection image contraband classification method based on information bottleneck according to claim 3 is characterized by: According to The predicted values ​​of all views of the k-th input image sample are used to make a comprehensive prediction, and finally the multi-view prediction output value is obtained. Calculate, then perform Hadamard product calculation on the calculation result and all views of the original k input image samples, and sum the calculation results to obtain the final prediction value.

5. The multi-view multi-label X-ray security inspection image contraband classification method based on information bottleneck according to claim 4 is characterized in that: Input the prepared training set into the classification model and perform multiple rounds of training until convergence; use the total loss function to measure the difference between the model's predicted output and the true label, minimize the loss function, and thus improve the model's prediction accuracy; During the training process, mAP is used as a metric to continuously monitor the performance of the model on the validation set. Whenever the best mAP value is obtained during training, the corresponding model configuration is recorded and saved as the optimal parameters.

Images