X-ray hazardous article detection method based on external memory enhanced visual language model

Abstract

The invention discloses an X-ray hazardous article detection method based on an external memory enhanced visual language model, and the method comprises the steps: S1, obtaining the original image data of a detected article, and carrying out the preprocessing of the data; s2, visual feature extraction is carried out on the preprocessed image data, patch embedding is generated through an encoder of a large visual model, and a patch embedding set is obtained; s3, carrying out low-rank compression on the attention weight, and carrying out similarity retrieval in a dangerous goods memory bank; s4, designing a small neural network; s5, current attention and historical attention are fused through confidence gating weighting; s6, performing element-by-element multiplication on the fused weight and the current image feature to obtain weighted feature representation; s7, inputting the weighted feature representation into two parallel prediction heads, and respectively executing classification and regression tasks; and S8, integrating the classification results generated by the two prediction heads and the coordinates of the detection frame, and carrying out visual display.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to an X-ray hazardous materials detection system based on an external memory-enhanced visual language model. Background Technology

[0002] In the field of customs dangerous goods inspection, X-ray imaging technology has become the mainstream detection method due to its non-invasiveness and high efficiency. However, traditional manual interpretation methods suffer from low efficiency and high false negative rates, while deep learning-based intelligent detection methods face challenges such as high computational resource consumption in complex scenarios and insufficient small-sample learning capabilities. In recent years, visual language models have shown certain advantages in object recognition tasks by integrating visual and semantic features, but their fixed parameter mechanism is difficult to adapt to the varied dangerous goods shapes and occlusion scenarios in X-ray images.

[0003] In existing technologies, Memory-Augmented Neural Networks (MANNs) enhance the long-term memory capability of models by introducing external storage modules (such as neural Turing machines and differentiable neural computers). However, these methods typically rely on end-to-end training memory matrices, making it difficult to scale to massive X-ray image data. RAMANN (Resistive Associative Memory-Augmented Neural Network) attempts to optimize memory access efficiency through in-memory computation, but it remains limited by hardware compatibility issues. Attention mechanisms (such as Transformers) improve feature extraction efficiency through dynamic weight allocation, but the computational complexity of standard self-attention increases quadratically with image resolution, leading to a significant increase in latency when processing high-resolution X-ray images. Improvements such as MAXFormer (Multi-Axis Attention eXchange Transformer) alleviate some problems through multi-scale feature fusion, but they do not address the redundant computation caused by insufficient reuse of historical attention patterns.

[0004] Therefore, current technology still has three major flaws: 1. Inefficient memory usage: Traditional MANNs end-to-end memory modules need to store all historical states, resulting in memory usage increasing linearly with task complexity; 2. Computational redundancy: The standard Transformer independently calculates attention weights for each X-ray image, ignoring the reusability of attention patterns for similar hazardous material images; 3. Insufficient dynamic adaptability: Existing systems have difficulty updating their memory contents in real time to adapt to new variants of hazardous materials. Summary of the Invention

[0005] The purpose of this invention is to provide an X-ray hazardous materials detection method based on an external memory-enhanced visual language model, so as to improve the detection and classification performance of hazardous materials in X-ray images, significantly improve computational efficiency and processing speed while ensuring detection accuracy, and reduce GPU memory usage.

[0006] The technical solution adopted in this invention is an X-ray hazardous materials detection method based on an external memory-enhanced visual language model, comprising the following steps: Step S1: Obtain the original image data of the item being detected and preprocess the data; Step S2: Visual features are extracted from the preprocessed image data, and patch embeddings are generated through the encoder of the large visual model to obtain a patch embedding set. Step S3: Perform low-rank compression on the attention weights and conduct a similarity search in the hazardous materials memory database; Step S4: Design a small neural network; Step S5: Based on a small neural network, current and historical attention are fused using confidence-gated weighted fusion. Step S6: Multiply the fused weights from S5 with the current image features element-wise to obtain the weighted feature representation; Step S7: Input the weighted feature representation obtained in S6 into two parallel prediction heads and perform classification and regression tasks respectively. Step S8: Integrate the classification results and bounding box coordinates generated by the two prediction heads and display them visually.

[0007] Furthermore, the specific steps of S3 are as follows: S31, based on the Transformer self-attention mechanism, linearly projects the patch embedding set into a query matrix and a key matrix respectively, calculates the multi-granularity relationship weights between suspicious regions and the global context in X-ray images, and represents them in a compact form, as shown in the following formula: in, Representing the query matrix Bond matrix The current relationship weight matrix between them It is a kernel function used to measure the similarity of vector pairs. It is a normalized activation function. Indicates a time step. These are temperature calibration parameters used to control the sharpness of the distribution. It is the core operator for performing low-rank approximation. It is a preset information fidelity constraint. The compact relation weight represents the current relation weight; S32, perform similarity retrieval based on a pre-set hazardous materials memory database, and assign weights to compact relationships. As a query probe, the formula is as follows: in, Compact relation weights representing the current relation weights Hazardous materials memory bank The matching score recorded in the middle, It is a function used to calculate the structural similarity between two relation weight matrices. This indicates the first item in the hazardous materials memory bank. 1 record This indicates the total quantity recorded in the hazardous materials memory bank. It is a sorting and filtering operator used to return the highest resonance score. A collection of historical experience records constitutes the search results set. .

[0008] Furthermore, the small neural network described in S4 The details are as follows: Input layer: Compact relational weights for the current time step and the set of search results retrieved from the hazardous materials memory database As input; Sequence modeling layer: The search results are arranged into a sequence in chronological order and input into a single-layer gated recurrent (such as RNN or LSTM) unit network to capture the temporal relationship of the above sequence; Gated output layer: The hidden state of the last time step of the sequence modeling layer is taken as the output, and this hidden state is fed into a fully connected layer, and a gating value is generated using the Sigmoid activation function.

[0009] Furthermore, in S5, the formula for weighted fusion of current and historical attention is as follows: in, It is a scalar confidence score with a range of 100%. , It is a small neural network defined by S4. It is a learnable parameter in a small neural network. It is a non-linear gated activation function. This indicates a splicing operation. Indicates the current attention weight. This indicates the historical attention weight retrieved from the hazardous materials memory bank. Indicates the weights after fusion. This represents element-wise multiplication. Indicates a time step. This indicates a vectorization operation.

[0010] Furthermore, in S6, the fused weights and current image features The formula for element-wise multiplication is as follows: in, It is a weighted feature representation. This indicates element-wise multiplication.

[0011] Furthermore, in S7, the classification head that performs the classification task is composed of a fully connected layer, and the input... It outputs the classification results of hazardous materials; the regression head that performs the regression task is also composed of fully connected layers, and the input... The specific location where the hazardous materials are exported.

[0012] The beneficial effects of this invention are 1. This invention uses low-rank compressed memory, which reduces memory usage by 38% compared to the traditional Transformer.

[0013] 2. The dynamic gating mechanism of this invention enables the secure reuse of historical attention weights, reducing redundant calculations by 30% to 50%.

[0014] 3. This invention has extremely high practical application effectiveness. In actual customs tests, it can improve the detection accuracy of overlapping items to 98.2%, and the single-frame processing latency is stable within 80ms, which can meet the millisecond-level response requirements of real-time security inspection. It also supports elastic deployment from edge devices to cloud clusters and can be adapted to application scenarios of different scales.

[0015] 4. This invention has scalability and flexibility. The federated memory architecture enables privacy-preserving sharing of cross-border customs data, and the incremental learning mechanism allows the system to learn about new dangerous goods with only 10 to 15 samples.

[0016] 5. The promotion and application of this invention will significantly improve security inspection efficiency, and is expected to reduce the workload of manual re-inspection by more than 70%, while reducing the rate of missed detection of dangerous goods to below 0.3%. Its technical framework can also be transferred to fields such as medical image analysis and industrial quality inspection, and has broad market prospects and social value. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the present invention.

[0019] Figure 2 This is a detailed architecture diagram of the memory-enhanced Transformer. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0021] Embodiments of the present invention provide an X-ray hazardous materials detection method based on an external memory-enhanced visual language model, the flowchart of which is shown below. Figure 1 As shown, the steps include: Step S1: Obtain the original image data of the item to be inspected using an X-ray scanning device, and preprocess the data to remove noise and improve image quality.

[0022] Step S2: Visual features are extracted from the preprocessed image data using a convolutional neural network, and patch embeddings are generated through the encoder of the large visual model to obtain a patch embedding set. ,in, Represents the set of real numbers. For the number of patches, For the embedded dimension.

[0023] Step S3 involves low-rank compression of the attention weights and similarity retrieval in the hazardous materials memory database to enhance the multi-scale attention perception mechanism for hazardous materials. The specific steps are as follows: S31, based on the Transformer self-attention mechanism, the architecture diagram of Transformer is as follows. Figure 2 As shown, the patch embedding set is linearly projected into a query matrix and a key matrix, respectively. The multi-granularity relationship weights between suspicious regions in X-ray images and the global context are calculated and then compacted, as shown in the following formula: in, Representing the query matrix Bond matrix The current relationship weight matrix between the suspected region and the global environment, i.e., the original multi-granularity relationship weight matrix between the suspected region and the global environment; It is a kernel function used to measure the similarity of vector pairs; It is a normalized activation function used to ensure the effectiveness of the weight distribution; Indicates a time step. These are temperature calibration parameters used to control the sharpness of the distribution. It is the core operator for performing low-rank approximation, used to compress the weights of the original relation. It is a preset information fidelity constraint, usually set to a value of the original dimension. , This represents the compact relation weight, which indicates the current relation weight.

[0024] S32, based on a preset hazardous materials memory database (which can be set according to different security inspection needs), performs similarity retrieval and assigns weights to compact relationships. As a query probe, the formula is as follows: in, Compact relation weights representing the current relation weights Hazardous materials memory bank The matching score recorded in the middle, It is a function used to calculate the structural similarity of two relation weight matrices. It is defined as the normalized inner product of the two matrices after normalization vectorization transformation, which can effectively resist slight rotation and scaling changes. This indicates the first item in the hazardous materials memory bank. 1 record This indicates the total quantity recorded in the hazardous materials memory bank. It is a sorting and filtering operator used to return the highest resonance score. A collection of historical experience records constitutes the search results set. .

[0025] Step S4: Design a small neural network This is used to capture temporal dependencies in historical hazardous materials patterns, as follows: Input layer: Compact relational weights for the current time step and the set of search results retrieved from the hazardous materials memory database As input; Sequence modeling layer: The search results are arranged into a sequence in chronological order and input into a single-layer gated recurrent (such as RNN or LSTM) unit network to capture the temporal relationship of the above sequence; Gated output layer: The hidden state of the last time step of the sequence modeling layer is taken as the output, and this hidden state is fed into a fully connected layer, and a gating value is generated using the Sigmoid activation function.

[0026] Step S5: Based on the small neural network designed in S4, the current and historical attention are fused using confidence-gated weighted fusion, as shown in the following formula: in, It is a scalar confidence score with a range of . This is used to dynamically balance the fusion ratio of current and historical attention weights; It is a small neural network defined by S4. It is a learnable parameter in a small neural network. It is a non-linear gated activation function. This indicates the operation of concatenating two vectorized attention values. Indicates the current attention weight. This indicates the historical attention weight retrieved from the hazardous materials memory bank. Indicates the weights after fusion. This represents element-wise multiplication. Indicates a time step. This indicates a vectorization operation.

[0027] Step S6: Combine the weights from S5 and current image features Element-wise multiplication yields the weighted feature representation, as shown in the following formula: in, It is a weighted feature representation. This indicates element-wise multiplication.

[0028] After this step, The suspicious regions are assigned high weights. Features in low-weight regions will be significantly enhanced, while features in low-weight regions will be suppressed. This is equivalent to providing the model with a new feature map that is double-calibrated by "historical experience" and "current observation" before making a final judgment.

[0029] Step S7, the weighted feature representation obtained in S6 Two parallel prediction heads are input, performing classification and regression tasks respectively. The classification head consists of fully connected layers. It outputs the classification results of hazardous materials (e.g., knives, liquid explosives, etc.); the regression head is also composed of fully connected layers, and the input... It outputs the specific location of the hazardous material, i.e., the coordinates of the detection frame.

[0030] Step S8: Integrate the classification results and bounding box coordinates generated by the two prediction heads, and visualize them through a human-computer interaction interface.

[0031] Experimental verification 1. High Similarity Item Detection (Efficiency and Accuracy Test): Taking the common charger detection scenario in customs security checks as an example, input image: Image 1: The suitcase contains a charger; Image 2: Another charger in a different suitcase (angle slightly different); Processing procedure: For image 1, the system fully calculates its attention weights and stores them in memory. For image 2: the system retrieves the storage pattern of image 1, with a similarity of 0.87. The gating mechanism assigns a weight of 0.65 to the historical pattern, requiring only the calculation of the remaining 35% of the attention. The final detection time is reduced from 120ms to 75ms, while maintaining an accuracy of 98.2%. A comparison of the performance of this invention and the traditional Transformer is shown in Table 1. Table 1. Comparison of the effects of the present invention and the traditional Transformer.

[0032] Second, detection under partial obstruction and complex background (robustness test): In actual security inspections, dangerous goods are often obscured by other items. This scenario aims to verify the robustness of the present invention in detecting incomplete information.

[0033] Test sample: A folding knife whose handle was partially obscured by multiple sets of keys and coins.

[0034] The processing logic of this invention is as follows: Even if the handle is obscured, the attention pattern of the exposed "blade" part can be matched with the historical record of the "knife" in the memory bank with a high score. The model completes the information of the obscured part by retrieving the complete "knife" pattern, thereby making the correct judgment.

[0035] Table 2 shows the comparison results of mAP between the present invention and other existing methods: Table 2. Comparison of mAP results between the present invention and traditional Transformer and Faster R-CNN. Analysis: In challenging occlusion scenarios, the mAP (mean Average Precision) of traditional methods drops significantly, while this invention demonstrates its superior robustness and adaptability to complex environments through association and completion capabilities.

[0036] III. Novel and Rare Item Detection (Generalization Ability Test): This scenario is used to test the model's ability to generalize detection of novel hazardous items (such as 3D-printed weapon parts) that have not appeared or have appeared very rarely in the training set.

[0037] Test sample: 3D printed gun lower receiver. Its overall structure did not appear in the training set, but the X-ray features of its internal components such as springs and latches are similar to common items (such as staplers and mechanical parts).

[0038] The processing logic of this invention is as follows: The model cannot match the entire item, but its attention mechanism can focus on local features such as "spring-like structures" and "high-density block structures," and retrieve the historical records corresponding to these local patterns in the memory bank. By fusing these "component-level" memories, although the model does not recognize the whole, it can determine that this is a "high-risk unknown item" composed of multiple suspicious parts, thereby issuing a warning.

[0039] Table 3 shows the comparison of recall rates between this invention and other existing methods: Table 3. Comparison of recall rates between the present invention and traditional Transformer and Faster R-CNN. Traditional methods are almost incapable of identifying new hazardous materials, while this invention improves recall by an order of magnitude through component-based associative reasoning, demonstrating its powerful generalization ability and potential to discover unknown threats.

[0040] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0041] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for detecting hazardous materials via X-ray based on an external memory-enhanced visual language model, characterized by the following steps: include: Step S1: Obtain the original image data of the item being detected and preprocess the data; Step S2: Visual features are extracted from the preprocessed image data, and patch embeddings are generated through the encoder of the large visual model to obtain a patch embedding set. Step S3: Perform low-rank compression on the attention weights and conduct a similarity search in the hazardous materials memory database; Step S4: Design a small neural network; Step S5: Based on a small neural network, current and historical attention are fused using confidence-gated weighted fusion. Step S6: Multiply the fused weights from S5 with the current image features element-wise to obtain the weighted feature representation; Step S7: Input the weighted feature representation obtained in S6 into two parallel prediction heads and perform classification and regression tasks respectively. Step S8: Integrate the classification results and bounding box coordinates generated by the two prediction heads and display them visually.

2. The X-ray hazardous materials detection method based on an external memory-enhanced visual language model according to claim 1, characterized in that, The specific steps of S3 are as follows: S31, based on the Transformer self-attention mechanism, linearly projects the patch embedding set into a query matrix and a key matrix respectively, calculates the multi-granularity relationship weights between suspicious regions and the global context in X-ray images, and represents them in a compact form, as shown in the following formula: in, Representing the query matrix Bond matrix The current relationship weight matrix between them It is a kernel function used to measure the similarity of vector pairs. It is a normalized activation function. Indicates a time step. These are temperature calibration parameters used to control the sharpness of the distribution. It is the core operator for performing low-rank approximation. It is a preset information fidelity constraint. The compact relation weight represents the current relation weight; S32, perform similarity retrieval based on a pre-set hazardous materials memory database, and assign weights to compact relationships. As a query probe, the formula is as follows: in, Compact relation weights representing the current relation weights Hazardous materials memory bank The matching score recorded in the middle, It is a function used to calculate the structural similarity between two relation weight matrices. This indicates the first item in the hazardous materials memory bank. 1 record This indicates the total quantity recorded in the hazardous materials memory bank. It is a sorting and filtering operator used to return the highest resonance score. A collection of historical experience records constitutes the search results set. .

3. The X-ray hazardous materials detection method based on an external memory-enhanced visual language model according to claim 1, characterized in that, The small neural network described in S4 The details are as follows: Input layer: Compact relational weights for the current time step and the set of search results retrieved from the hazardous materials memory database As input; Sequence modeling layer: The search results are arranged into a sequence in chronological order and input into a single-layer gated recurrent unit network to capture the temporal relationship of the above sequence; Gated output layer: The hidden state of the last time step of the sequence modeling layer is taken as the output, and this hidden state is fed into a fully connected layer, and a gating value is generated using the Sigmoid activation function.

4. The X-ray hazardous materials detection method based on an external memory-enhanced visual language model according to claim 1, characterized in that, In S5, the formula for weighted fusion of current and historical attention is as follows: in, It is a scalar confidence score with a range of 100%. , It is a small neural network defined by S4. It is a learnable parameter in a small neural network. It is a non-linear gated activation function. This indicates a splicing operation. Indicates the current attention weight. This indicates the historical attention weight retrieved from the hazardous materials memory bank. Indicates the weights after fusion. This represents element-wise multiplication. Indicates a time step. This indicates a vectorization operation.

5. The X-ray hazardous materials detection method based on an external memory-enhanced visual language model according to claim 1, characterized in that, In S6, the fused weights and current image features The formula for element-wise multiplication is as follows: in, It is a weighted feature representation. This indicates element-wise multiplication.

6. The X-ray hazardous materials detection method based on an external memory-enhanced visual language model according to claim 1, characterized in that, In step S7, the classification head that performs the classification task is composed of a fully connected layer, and the input... It outputs the classification results of hazardous materials; the regression head that performs the regression task is also composed of fully connected layers, and the input... The specific location where the hazardous materials are exported.

Images