Article detection method and article detection device for security images

Abstract

The application discloses an article detection method and device for security inspection images, and relates to the technical field of computer vision and intelligent security inspection, and the method comprises the following steps: inputting a security inspection image into a front-end lightweight detection network model, wherein the front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy, and the front-end lightweight detection network model comprises a feature extractor and a neck network; screening a first type of candidate frame, inputting an image block into a rear-end multi-modal large model, identifying the category, morphological features and attribute information of suspicious contraband in the image block by the rear-end multi-modal large model, and generating natural language text; for a second type of candidate frame, generating structured detection information, aggregating the natural language text and the structured detection information, and outputting an article security inspection report. The application solves the technical problem that a target detection algorithm based on deep learning is easy to ignore small target features and has a high missing detection rate in a security inspection scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent security inspection technology, and more specifically, to a method and device for detecting items in security inspection images. Background Technology

[0002] Airline baggage security screening is a crucial link in ensuring civil aviation safety, and its efficiency and accuracy directly affect airport throughput and public safety. Currently, X-ray security image detection of prohibited items mainly relies on manual interpretation or traditional automated auxiliary detection systems. However, with the continuous growth of global passenger traffic and the diversification and concealment of prohibited items, existing technologies are gradually revealing many limitations when facing complex security screening scenarios.

[0003] In the security inspection process of related technologies, security personnel need to stare at X-ray screens for extended periods to manually identify items inside luggage. Due to the characteristics of X-ray images, such as perspective overlay and complex color coding (organic, inorganic, and mixed substances), and the frequent occurrence of piled-up luggage and severe obstruction, security personnel are prone to visual fatigue and decreased attention. This not only results in a high rate of missed detections (especially for small prohibited items such as miniature knives and concealed batteries), but also makes it difficult to ensure consistency and objectivity in the inspection process due to differences in interpretation standards among different security personnel.

[0004] Another detection method, deep learning-based object detection algorithms, is widely used in scenarios such as airports and subways where baggage inspection is required. However, these convolutional neural network-based algorithms mainly rely on local receptive fields and have inherent shortcomings in capturing global contextual information of images. In X-ray images, contraband (such as chips, micro batteries, etc.) often occupy a very small proportion of pixels and are easily confused with background clutter. Traditional object detection algorithms tend to ignore these small target features. At the same time, the occlusion problem is serious, and there are often multiple overlapping items in the baggage, resulting in incomplete target features and easy inaccurate detection box localization or missed detection.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a method and apparatus for detecting objects in security inspection images, which at least solves the technical problem that related technologies using deep learning-based target detection algorithms in security inspection scenarios tend to overlook small target features and have a high false negative rate.

[0007] According to one aspect of the present invention, a method for detecting items in security inspection images is provided, comprising: inputting at least one acquired security inspection image into a front-end lightweight detection network model, wherein the front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy, the front-end lightweight detection network model includes: a feature extractor and a neck network, the feature extractor being used to extract global multi-scale features of the security inspection image to focus on suspicious contraband areas, the neck network being used to perform target perception on the suspicious contraband areas, and outputting multiple candidate boxes containing image patches and the confidence scores of each candidate box; filtering first-class candidate boxes with confidence scores below a preset threshold, inputting the image patches of the first-class candidate boxes into a back-end multimodal large model, and having the back-end multimodal large model identify the category, morphological features, and attribute information of the suspicious contraband in the image patches, generating natural language text; generating structured detection information for second-class candidate boxes with confidence scores greater than or equal to a preset threshold; and aggregating the natural language text of the image patches corresponding to the first-class candidate boxes and the structured detection information of the second-class candidate boxes to output an item security inspection report.

[0008] Optionally, the feature extractor includes: a visual backbone network based on the Transformer architecture, wherein the visual backbone network introduces a hierarchical feature extraction mechanism and a moving window self-attention mechanism. The hierarchical feature extraction mechanism is used to progressively downsample the security inspection image by stacking multiple stages to generate a global multi-scale feature map. The moving window self-attention mechanism is used to divide the security inspection image into non-overlapping windows, calculate the self-attention mechanism within each window, and offset the window division to allow information interaction between different windows, so as to initially screen suspicious regions of the security inspection image; a convolutional block attention unit, including a channel attention module and a spatial attention module, wherein the channel attention module is used to aggregate global information of the security inspection image to recalibrate the weights of each channel to obtain a feature map, and each channel is responsible for detecting different image features; the spatial attention module is used to focus on the spatial position in the feature map, and combined with the initial screening of suspicious regions of the security inspection image, output a spatial attention map containing the suspicious regions.

[0009] Optionally, the neck network includes a feature pyramid network. After the spatial attention map is input into the feature pyramid network, multi-scale feature fusion is performed by the feature fusion path of the feature pyramid network, and the detection head outputs the bounding box coordinates and corresponding confidence scores of all suspected contraband areas.

[0010] Optionally, training the lightweight front-end detection network model includes: integrating security inspection images from multi-source public datasets, obtaining actual images of items passed through security checkpoints collected by collaborating partners through a federated learning framework, desensitizing the actual images, and then performing image annotation processing to construct a security inspection image dataset containing preset annotation information. The security inspection image dataset contains multiple dual-view images, each containing various targets to be identified. The security inspection image dataset is divided into a training set, a validation set, and a test set at a fixed ratio. Random data augmentation processing is applied to the training set, and multiple pairs of text pairs describing security inspection images and contraband are constructed. The data augmentation processing includes horizontal flipping, color adjustment, and noise addition. After initializing the architecture of the lightweight front-end detection network model, training is performed using the training set, and a composite loss function is used for evaluation. After each training epoch, the loss is calculated on the validation set, and the performance parameters of the lightweight front-end detection network model are monitored. If the performance parameters meet the requirements, training is stopped, resulting in the trained lightweight front-end detection network model.

[0011] Optionally, the composite loss function includes: a coordinate loss sub-function, used to define the regression error of the center point and width and height of the predicted candidate box; an intersection-union ratio (IoU) loss sub-function, used to optimize the intersection-union ratio between the predicted candidate box and the ground truth box; a confidence loss sub-function, used to associate the confidence of the candidate box output by the front-end lightweight detection network model with the actual localization accuracy using binary cross-entropy, with the target confidence value being the intersection-union ratio between the predicted candidate box and the ground truth box; and a classification loss sub-function, which uses single-class foreground binary cross-entropy loss for the detection of contraband of a specific class.

[0012] Optionally, the hybrid granularity pruning strategy includes: using the L1 norm as a criterion, globally ranking the weights of all two-dimensional convolutional layers and fully connected layers in the front-end lightweight detection network model by importance, pruning weights with importance lower than a preset importance threshold to complete global unstructured pruning; and on the basis of global unstructured pruning, structurally removing redundant convolutional channels to complete local channel-level structured pruning.

[0013] Optionally, the backend multimodal large model is adjusted by injecting a low-rank adapter into the attention module of the backend multimodal large model, and using the low-rank adapter to fine-tune the model using a pre-built image-text pairing dataset.

[0014] Optionally, before inputting at least one acquired security inspection image into the front-end lightweight detection network model, the method further includes: acquiring at least one original security inspection image after the item enters the security inspection station or security inspection channel, wherein the type of the security inspection image includes at least: an X-ray image; performing standardized preprocessing on the original security inspection image to generate the security inspection image in a uniform format, wherein the standardized preprocessing includes at least one of the following: image size normalization, noise removal, and contrast enhancement.

[0015] According to another aspect of the present invention, a method for detecting items in security inspection images is also provided, comprising: inputting at least one acquired X-ray security inspection image into a front-end lightweight detection network model, wherein the front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy, wherein a visual backbone network based on a Transformer architecture is used for feature extraction in the front-end lightweight detection network, and the extracted global multi-scale features are input into a convolutional attention unit to focus on the region of suspected contraband items, multi-scale fusion is performed using a neck network, and a detection head is used to output multiple candidate boxes containing image patches and the confidence scores of each candidate box; for confidence scores greater than or equal to a pre-defined value, the method further comprises: inputting at least one acquired X-ray security inspection image into a front-end lightweight detection network model, wherein the front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy, wherein a visual backbone network based on a Transformer architecture is used for feature extraction, and the extracted global multi-scale features are input into a convolutional attention unit to focus on the region of suspected contraband items, and a neck network is used for multi-scale fusion, and a detection head is used to output multiple candidate boxes containing image patches and the confidence scores of each candidate box; for candidate boxes with confidence scores greater than or equal to a pre-defined value, the method further comprises: inputting at least one acquired X-ray security inspection image into a front-end lightweight detection network model, wherein the front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy, wherein a visual backbone network based on a Transformer architecture is used for feature extraction ... Candidate boxes with a set confidence threshold are marked as high-confidence results. The bounding box coordinates, confidence values, category information, and morphological features of the suspected contraband are retained to obtain structured detection information. For candidate boxes with confidence values ​​less than the preset confidence threshold, they are marked as low-confidence results. The image blocks corresponding to the candidate boxes of the low-confidence results are input into the backend multimodal large model. The backend multimodal large model identifies the category, morphological features, and attribute information of the suspected contraband in the image blocks and generates natural language text. The natural language text of the image blocks corresponding to the low-confidence results and the structured detection information of the image blocks corresponding to the high-confidence results are aggregated to output an item security inspection report.

[0016] According to another aspect of the present invention, an object detection device for security inspection images is also provided, comprising: a security inspection image input unit for inputting at least one acquired security inspection image into a front-end lightweight detection network model, wherein the front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy, the front-end lightweight detection network model includes: a feature extractor and a neck network, the feature extractor being used to extract global multi-scale features of the security inspection image to focus on suspicious contraband areas, the neck network being used to perform target perception on the suspicious contraband areas, and outputting multiple candidate boxes containing image patches and the confidence scores of each candidate box; and a back-end large-scale model. The system includes a type recognition unit for filtering first-type candidate boxes with confidence levels below a preset threshold, inputting the image blocks of these first-type candidate boxes into a back-end multimodal large model, which then identifies the category, morphological features, and attribute information of the suspected contraband in the image blocks and generates natural language text. A high-confidence candidate box information generation unit is used to generate structured detection information for second-type candidate boxes with confidence levels greater than or equal to a preset threshold. An item security inspection report output unit aggregates the natural language text of the image blocks corresponding to the first-type candidate boxes and the structured detection information of the second-type candidate boxes to output an item security inspection report.

[0017] Optionally, the feature extractor includes: a visual backbone network based on the Transformer architecture, wherein the visual backbone network introduces a hierarchical feature extraction mechanism and a moving window self-attention mechanism. The hierarchical feature extraction mechanism is used to progressively downsample the security inspection image by stacking multiple stages to generate a global multi-scale feature map. The moving window self-attention mechanism is used to divide the security inspection image into non-overlapping windows, calculate the self-attention mechanism within each window, and offset the window division to allow information interaction between different windows, so as to initially screen suspicious regions of the security inspection image; a convolutional block attention unit, including a channel attention module and a spatial attention module, wherein the channel attention module is used to aggregate global information of the security inspection image to recalibrate the weights of each channel to obtain a feature map, and each channel is responsible for detecting different image features; the spatial attention module is used to focus on the spatial position in the feature map, and combined with the initial screening of suspicious regions of the security inspection image, output a spatial attention map containing the suspicious regions.

[0018] Optionally, the neck network includes a feature pyramid network. After the spatial attention map is input into the feature pyramid network, multi-scale feature fusion is performed by the feature fusion path of the feature pyramid network, and the detection head outputs the bounding box coordinates and corresponding confidence scores of all suspected contraband areas.

[0019] Optionally, the item detection device for security inspection images, when training the front-end lightweight detection network model, includes: an image integration unit, which integrates security inspection images from multi-source public datasets and obtains actual images of items passing through security checkpoints collected by collaborating partners through a federated learning framework; after desensitizing the actual images, it performs image annotation processing to construct a security inspection image dataset containing preset annotation information, wherein the security inspection image dataset contains multiple dual-view images, and each security inspection image contains various types of targets to be identified; and a dataset partitioning unit, which divides the security inspection image dataset into a training set according to a fixed ratio. The system comprises a validation set and a test set, and performs random data augmentation on the training set, constructing multiple pairs of text descriptions of security inspection images and prohibited items. The data augmentation includes horizontal flipping, color adjustment, and noise addition. A model training unit is used to train the lightweight front-end detection network model using the training set after initializing the architecture of the model. A composite loss function is used for evaluation. After each training epoch, the loss is calculated on the validation set, and the performance parameters of the lightweight front-end detection network model are monitored. Training is stopped when the performance parameters meet the requirements, resulting in the trained lightweight front-end detection network model.

[0020] Optionally, the composite loss function includes: a coordinate loss sub-function, used to define the regression error of the center point and width and height of the predicted candidate box; an intersection-union ratio (IoU) loss sub-function, used to optimize the intersection-union ratio between the predicted candidate box and the ground truth box; a confidence loss sub-function, used to associate the confidence of the candidate box output by the front-end lightweight detection network model with the actual localization accuracy using binary cross-entropy, with the target confidence value being the intersection-union ratio between the predicted candidate box and the ground truth box; and a classification loss sub-function, which uses single-class foreground binary cross-entropy loss for the detection of contraband of a specific class.

[0021] Optionally, the hybrid granularity pruning strategy includes: using the L1 norm as a criterion, globally ranking the weights of all two-dimensional convolutional layers and fully connected layers in the front-end lightweight detection network model by importance, pruning weights with importance lower than a preset importance threshold to complete global unstructured pruning; and on the basis of global unstructured pruning, structurally removing redundant convolutional channels to complete local channel-level structured pruning.

[0022] Optionally, the item detection device for security inspection images includes, when adjusting the back-end multimodal large model: a low-rank adapter injection module, for injecting a low-rank adapter into the attention module of the back-end multimodal large model, and using the low-rank adapter to fine-tune the model using a pre-built image-text pairing dataset.

[0023] Optionally, the item detection device for security inspection images further includes: a security inspection image acquisition unit, used to acquire at least one original security inspection image after the item enters the security inspection station or security inspection channel before inputting at least one acquired security inspection image into the front-end lightweight detection network model, wherein the type of the security inspection image includes at least: an X-ray image; and an image preprocessing unit, used to perform standardized preprocessing on the original security inspection image to generate the security inspection image in a uniform format, wherein the standardized preprocessing includes at least one of the following: image size normalization, noise removal, and contrast enhancement.

[0024] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform any of the above-described methods for detecting items in security inspection images.

[0025] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the item detection method for security inspection images described above.

[0026] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the item detection method for security inspection images described in any one of the above embodiments.

[0027] In this disclosure, at least one security inspection image is input into a front-end lightweight detection network model. This model is obtained through a hybrid granularity pruning strategy and includes a feature extractor and a neck network. The feature extractor extracts global multi-scale features from the security inspection image to focus on suspicious contraband areas. The neck network performs target perception on these areas, outputting multiple candidate boxes containing image patches and the confidence score of each box. First-class candidate boxes with confidence scores below a preset threshold are selected, and the image patches containing these first-class candidate boxes are input into a back-end multimodal large-scale model. The back-end multimodal large-scale model identifies the category, morphological features, and attribute information of the suspicious contraband in the image patches, generating natural language text. For second-class candidate boxes with confidence scores greater than or equal to a preset threshold, structured detection information for these second-class candidate boxes is generated. Finally, the natural language text of the image patches corresponding to the first-class candidate boxes and the structured detection information of the second-class candidate boxes are aggregated to output an item security inspection report.

[0028] In this disclosure, a feature extractor (using a Transformer architecture) and a neck network can be used to effectively detect small and occluded targets in security inspection images. Simultaneously, a hybrid granularity pruning strategy is employed to reduce the size of the front-end lightweight detection network model, meeting the real-time requirements of edge devices and shortening inference time. Furthermore, this invention utilizes a large-model collaborative mechanism of front-end lightweight detection and back-end multimodal large-model analysis to perform deep semantic analysis and text description generation on low-confidence candidate regions, significantly reducing the false negative rate and improving the interpretability of the detection. This addresses the technical problem in security inspection scenarios where deep learning-based target detection algorithms tend to overlook small target features, resulting in a high false negative rate. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0030] Figure 1 This is a flowchart of an optional item detection method for security inspection images according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of an optional security inspection image contraband detection system based on size model collaboration according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of an optional item detection device for security inspection images according to an embodiment of the present invention;

[0033] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) that performs an item detection method for security inspection images according to an embodiment of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0037] The Convolutional Block Attention Module (CBAM) is a lightweight attention mechanism module consisting of channel attention and spatial attention. It is used to adaptively recalibrate the channel weights and spatial locations of feature maps, focusing on key regions to improve the perception of occlusions and small objects.

[0038] Feature Pyramid Network (FPN) is a multi-scale feature fusion network that combines high-level semantic information with low-level detailed information through top-down paths and lateral connections to generate multi-scale feature maps with strong semantic expressive power, supporting the detection of targets of different sizes.

[0039] Intersection over Union (IoU) is a metric that measures the degree of overlap between the predicted bounding box and the ground truth bounding box. It is defined as the ratio of the intersection area to the union area of ​​the two boxes and is used to evaluate the localization accuracy of object detection.

[0040] Low-Rank Adaptation (LoRA) is an efficient technique for fine-tuning large models. It updates parameters by injecting a low-rank matrix adapter next to the weights of the pre-trained model. With the base model parameters frozen, it adapts the model to specific tasks, such as X-ray image description generation, at extremely low computational cost.

[0041] You Only Look Once, or YOLO for short, is a single-stage object detection algorithm paradigm. Its detection head directly predicts bounding boxes and confidence scores from feature maps, and it features fast inference speed and suitability for real-time deployment.

[0042] The AdamW optimizer, or AdamW for short, is a stochastic gradient descent optimization algorithm based on adaptive moment estimation and weight decay. It is used for parameter updates during model training and helps the model converge faster and more stably.

[0043] Cosine Annealing Learning Rate Scheduler, or Cosine Annealing for short, is a learning rate adjustment strategy that periodically or monotonically reduces the learning rate according to the cosine function curve, helping the model to adjust parameters more finely in the later stages of training and avoid getting trapped in local optima.

[0044] Binary Cross-Entropy Loss (BCE Loss) is a loss function used for binary classification problems. It measures the difference between the predicted probability distribution and the true label distribution and is often used for classification tasks and the optimization of confidence supervision signals.

[0045] L1 norm, or simply L1 norm, is the sum of the absolute values ​​of vector or matrix elements. It is used to evaluate the importance of model parameters and is often used as a criterion in the pruning process to identify and remove redundant weights with small absolute values ​​that contribute little to the output.

[0046] It should be noted that the item detection method and apparatus for security inspection images disclosed herein can be used in the fields of computer vision and intelligent security inspection technology, specifically in the case of detecting prohibited items in security inspection images based on visual feature extractors and multimodal large models. It can also be used in any field other than computer vision and intelligent security inspection technology. In the case of detecting prohibited items in security inspection images based on visual feature extractors and multimodal large models, this disclosure does not limit the application area of ​​the item detection method and apparatus for security inspection images.

[0047] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data, etc.) collected in this public disclosure are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data are handled with corresponding access points, allowing users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0048] It should be noted that in this disclosure, customer information is collected and analyzed, and users are provided with corresponding operation entry points to choose whether to agree to or reject the automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0049] The following embodiments of the present invention can be applied to various systems / applications / equipment used for item detection in security screening images. The present invention is applicable to locations requiring security checks of baggage, parcels, or cargo, such as civil aviation airports, subway stations, train stations, and ports.

[0050] This invention achieves a balance between high precision and lightweight design. By using a feature extractor (adopting a Transformer architecture), a convolutional attention mechanism module, and hybrid granularity pruning, it effectively solves the problem of detecting small and occluded targets in security inspection images, while meeting the real-time requirements of edge devices, with an average inference time of ≤500 milliseconds.

[0051] This invention can improve the system's ability to process difficult samples. Through a collaborative mechanism of lightweight front-end detection and multimodal large model analysis at the back-end, it performs deep semantic analysis and text description generation on low-confidence candidate regions, significantly reducing the false negative rate and improving the interpretability of the detection.

[0052] This invention can optimize detection accuracy by using an IoU-aware confidence supervision method and a composite loss function to solve the misalignment problem between bounding box regression and classification confidence, thereby improving the localization accuracy of detection boxes and the reliability of model evaluation.

[0053] The present invention will now be described in detail with reference to various embodiments.

[0054] Example 1

[0055] According to an embodiment of the present invention, an embodiment of a method for detecting items in security inspection images is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0056] Figure 1 This is a flowchart of an optional item detection method for security inspection images according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0057] Step S101: Input at least one security inspection image into the front-end lightweight detection network model. The front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy. The front-end lightweight detection network model includes a feature extractor and a neck network. The feature extractor is used to extract global multi-scale features of the security inspection image to focus on the suspicious contraband area. The neck network is used to perform target perception on the suspicious contraband area and output multiple candidate boxes containing image patches and the confidence of each candidate box.

[0058] Optionally, before inputting at least one acquired security inspection image into the front-end lightweight detection network model, the method further includes: acquiring at least one original security inspection image after the item enters the security inspection station or security inspection channel, wherein the type of security inspection image includes at least: X-ray image; performing standardized preprocessing on the original security inspection image to generate a security inspection image in a uniform format, wherein the standardized preprocessing includes at least one of the following: image size normalization, noise removal, and contrast enhancement.

[0059] In this embodiment, the security inspection images include at least X-ray images. X-ray images penetrate luggage and reveal their internal structure by displaying different colors and textures based on material density and atomic number. Acquiring raw security inspection images helps obtain detailed information about the luggage's interior. The raw security inspection images can then undergo standardized preprocessing. Image size normalization adjusts raw images of different resolutions to a fixed size required for model input, helping to maintain consistency in the model's input dimensions and facilitating batch processing. Noise removal suppresses electronic or scattering noise generated during X-ray imaging, improving the image signal-to-noise ratio and reducing interfering features. Contrast enhancement highlights the grayscale differences between different materials inside the luggage, enhancing the visual contrast between contraband and background clutter. This allows the feature extractor to more clearly extract the texture and contour features of suspicious areas, thereby improving the detection accuracy and robustness of the lightweight front-end detection network model.

[0060] Optionally, the feature extractor includes: a visual backbone network based on the Transformer architecture, wherein the visual backbone network introduces a hierarchical feature extraction mechanism and a moving window self-attention mechanism. The hierarchical feature extraction mechanism is used to progressively downsample the security inspection image by stacking multiple stages to generate a global multi-scale feature map. The moving window self-attention mechanism is used to divide the security inspection image into non-overlapping windows, calculate the self-attention mechanism within each window, and offset the window division to allow information interaction between different windows, so as to initially screen suspicious regions of the security inspection image; a convolutional block attention unit, including a channel attention module and a spatial attention module, wherein the channel attention module is used to aggregate global information of the security inspection image to recalibrate the weights of each channel to obtain a feature map, and each channel is responsible for detecting different image features; the spatial attention module is used to focus on the spatial position in the feature map, and combined with the initial screening of suspicious regions of the security inspection image, outputs a spatial attention map containing the suspicious regions.

[0061] The visual backbone network introduces a hierarchical feature extraction mechanism and a moving window self-attention mechanism. The hierarchical feature extraction mechanism captures semantic information at different scales, from low-level edge textures to high-level object structures, providing a rich feature base for subsequent multi-scale detection. The moving window self-attention mechanism helps to effectively capture the global contextual dependencies of the image while reducing computational complexity, thereby initially screening suspicious areas in security inspection images and improving the ability to perceive target features in complex backgrounds. The feature extractor also includes a convolutional block attention unit, consisting of a channel attention module and a spatial attention module. Each channel in the channel attention module is responsible for detecting different image features. Through this module, the model can adaptively emphasize feature channels that are more valuable for identifying contraband (such as lithium batteries, sharp objects, etc.) and suppress interference from irrelevant background channels, allowing the model to focus more on discriminative semantic information. Furthermore, the spatial attention module focuses on the spatial location in the feature map, helping the model locate the specific distribution area of ​​contraband in the image. This spatial focusing mechanism helps refine the detection range, especially when dealing with severe occlusion or small targets. It guides the network to focus on key spatial locations, thereby assisting the subsequent detection head in more accurately locating the boundaries of contraband and improving the regression accuracy of the detection boxes.

[0062] Optionally, the neck network includes a feature pyramid network. After the spatial attention map is input into the feature pyramid network, multi-scale feature fusion is performed by the feature fusion path of the feature pyramid network, and the detection head outputs the bounding box coordinates and corresponding confidence scores of all suspected contraband areas.

[0063] After the spatial attention map is input into the Feature Pyramid Network (FPN), multi-scale feature fusion can be performed by the feature fusion path of the Feature Pyramid Network. The Feature Pyramid Network fuses high-level semantic information with low-level spatial information through top-down paths and lateral connections. This multi-scale feature fusion mechanism can enhance the model's ability to perceive prohibited items of different sizes (such as large luggage items and small batteries, knives, etc.), and combine the detailed features of shallow networks with the semantic features of deep networks to generate multi-scale feature maps with strong semantic expressive power.

[0064] Furthermore, the detection head outputs the bounding box coordinates and corresponding confidence scores of all suspected contraband areas. The detection head can adopt a YOLO-style architecture to directly predict targets on multi-scale feature maps. This detection head helps to output multiple candidate boxes containing image patches and the confidence scores of each candidate box. The bounding box coordinates are used to locate the contraband position, and the confidence score is used to characterize the probability that a contraband exists at that position, enabling fast inference and meeting the real-time requirements of security inspection scenarios.

[0065] Optionally, the training of the lightweight front-end detection network model includes: integrating security inspection images from multi-source public datasets, obtaining actual images of items passed through security checkpoints collected by collaborating partners through a federated learning framework, desensitizing the actual images, and then performing image annotation to construct a security inspection image dataset containing preset annotation information. This dataset contains multiple dual-view images, each containing various targets to be identified. The security inspection image dataset is divided into training, validation, and test sets at a fixed ratio. Random data augmentation is performed on the training set, and multiple pairs of text describing security inspection images and prohibited items are constructed. Data augmentation includes horizontal flipping, color adjustment, and noise addition. After initializing the architecture of the lightweight front-end detection network model, it is trained using the training set and evaluated using a composite loss function. After each training epoch, the loss is calculated on the validation set, and the performance parameters of the lightweight front-end detection network model are monitored. Training is stopped when the performance parameters meet the requirements, resulting in the trained lightweight front-end detection network model.

[0066] In this embodiment, when training the lightweight detection network model, security inspection images from multiple publicly available datasets are first integrated. These datasets can include, but are not limited to, SIXray, OPIXray, and HiXray. Integrating these datasets enriches the diversity of the training data, covering various types of contraband scenarios. Furthermore, this embodiment uses a federated learning framework to obtain actual images of items passed through security checkpoints collected by collaborating partners. The federated learning framework helps to compensate for insufficient samples in publicly available datasets (such as fire types and lithium batteries) by utilizing data from security checkpoints distributed across different geographical locations, while protecting data privacy. This improves the model's adaptability to real-world, complex scenarios. Optionally, after desensitizing the actual images, image annotation is performed. Desensitization helps remove parts of the images that involve personal privacy or sensitive information, meeting data compliance requirements. Annotation can be performed by a professional to annotate the bounding boxes and categories of contraband in the images. Pre-set annotation information includes bounding box coordinates and category labels. The constructed security inspection image dataset contains multiple dual-view images. Each security inspection image contains various targets that need to be identified. The dual-view images help the model learn the features of objects from different angles, improve the robustness of detection, enable the model to more comprehensively understand the morphology and structure of contraband, and help improve detection accuracy.

[0067] Furthermore, the security inspection image dataset is divided into training, validation, and test sets in a fixed ratio (e.g., 7:2:1). The training set is used for model parameter updates, the validation set for hyperparameter tuning and early stopping detection, and the test set for final performance evaluation. This helps prevent model overfitting and objectively assesses the model's generalization ability. Random data augmentation is performed on the training set. Horizontal flipping increases sample diversity and simulates targets with different orientations; color adjustment simulates image differences under different lighting or imaging conditions; and noise addition improves the model's robustness to image noise, expands the coverage of training data, and enhances the model's generalization performance in complex and variable environments. Additionally, this embodiment can construct multiple pairs of text descriptions of security inspection images and contraband. These text pairs can be used for fine-tuning of the backend multimodal large model, but constructing them at this stage helps clarify the correspondence between the frontend detection target and the semantic description, providing a data foundation for subsequent collaboration between large and small models.

[0068] Model initialization includes weight initialization and hyperparameter settings. For example, using the AdamW optimizer, a composite loss function is employed for evaluation. This composite loss function consists of a weighted average of coordinate loss, IoU loss, confidence loss, and classification loss. The coordinate loss defines the regression error of the predicted bounding box's center point and width / height; the IoU loss optimizes the intersection-over-union ratio (IoU) between the predicted and ground truth bounding boxes; the confidence loss uses a binary cross-entropy loss with a target value that is dynamically calculated as the IoU value—an IoU-aware strategy; and the classification loss uses a single-class foreground binary cross-entropy loss. This composite loss function design can address the misalignment between bounding box regression and classification confidence, enabling the model to improve localization accuracy while optimizing classification accuracy.

[0069] Optionally, after each training epoch, the loss is calculated on the validation set, and the performance parameters of the lightweight front-end detection network model are monitored. These performance parameters include metrics such as accuracy, recall, and mAP (mean precision) on the validation set. Training is stopped when the performance parameters meet the requirements, resulting in the trained lightweight front-end detection network model. Meeting the requirements may include the validation set loss no longer decreasing significantly, performance metrics reaching a preset threshold, or no performance improvement for several consecutive epochs. Stopping training helps prevent overfitting, preserves the optimal model parameters, and enables the model to initially screen for contraband in security inspection images.

[0070] Optionally, the composite loss function includes: a coordinate loss sub-function, used to define the regression error of the predicted candidate box center point and width and height; an intersection-union ratio (IoU) loss sub-function, used to optimize the intersection-union ratio between the predicted candidate box and the ground truth box; a confidence loss sub-function, used to correlate the confidence of the candidate box output by the lightweight detection network model with the actual localization accuracy using binary cross-entropy, with the target confidence value being the intersection-union ratio between the predicted candidate box and the ground truth box; and a classification loss sub-function, which uses single-class foreground binary cross-entropy loss for the detection of contraband of a specific class.

[0071] The coordinate loss function quantifies the localization bias by calculating the differences between the center coordinates, width, and height of the predicted bounding box and their corresponding values ​​in the ground truth bounding box. This allows the model to learn more accurate bounding box regressions, enabling the predicted boxes to more closely fit the actual contours of the target object, thereby improving the spatial localization accuracy of the detected boxes. The intersection-union ratio (IU) loss function optimizes the IU ratio between the predicted candidate boxes and the ground truth bounding boxes. The IU ratio measures the degree of overlap between the predicted and ground truth boxes. This function guides the model to optimize the overlapping area between the predicted and ground truth boxes by maximizing the IU ratio or minimizing its difference from 1, improving the overall coverage quality of the detected boxes, reducing deviations of the predicted boxes from the actual target, and thus improving the localization performance of target detection. The confidence loss function implements an IoU-aware confidence supervision strategy, allowing the model to learn "localization accuracy" while determining "target presence." Through this correlation, the model links the confidence score with localization quality, meaning that high confidence not only indicates a high probability of target presence but also suggests higher localization accuracy, thus solving the problem of misalignment between confidence and localization accuracy in traditional detection. The classification loss function, for the detection of prohibited items of a specific category, distinguishes between the foreground target (i.e., the prohibited item) and the background. By calculating the binary cross-entropy error between the predicted category probability and the true label, the model can focus on the identification of prohibited item categories, enhance the model's ability to distinguish the characteristics of specific prohibited items such as fire types, lithium batteries, and sharp objects, and improve the classification accuracy.

[0072] Optionally, the hybrid granularity pruning strategy includes: using the L1 norm as a criterion, globally ranking the weights of all two-dimensional convolutional layers and fully connected layers in the front-end lightweight detection network model by importance, and pruning weights with importance below a preset importance threshold to complete global unstructured pruning; based on global unstructured pruning, structurally removing redundant convolutional channels to complete local channel-level structured pruning.

[0073] The L1 norm assesses parameter importance by calculating the sum of the absolute values ​​of weights. Weights with smaller values ​​are considered to contribute less to the model output. Weights with importance below a preset importance threshold are pruned, and a large number of tiny weight values ​​in the model are set to zero, significantly reducing the number of model parameters, breaking the structural constraints of weights, and reducing the model's storage requirements. Furthermore, redundant convolutional channels can be structurally removed. This involves identifying and deleting convolutional channels that, after global pruning, contribute little to the overall response of the feature map. This structural pruning helps restore the sparsity structure of the model on hardware, enabling the model to utilize existing sparse computing libraries or hardware acceleration features to further reduce computational complexity. By removing redundant channels, the amount of memory access and floating-point operations during inference are reduced, thereby improving the model's operating efficiency on edge devices, achieving faster inference speeds, and reducing power consumption.

[0074] Step S102: Select the first type of candidate boxes corresponding to confidence scores below a preset threshold, input the image block of the first type of candidate box into the backend multimodal large model, and the backend multimodal large model identifies the category, morphological features and attribute information of the suspected contraband in the image block to generate natural language text.

[0075] The confidence level represents the probability of the presence of contraband. When the confidence level is lower than a preset threshold, it indicates that the region corresponding to the candidate box has low confidence and may contain contraband, or that the target features are complex or occluded, making it difficult for the front-end model to determine accurately. Filtering out these first-class candidate boxes allows computational resources to be concentrated on difficult samples, avoiding redundant large-model inference on high-confidence and certain targets. The back-end multimodal large model, based on its pre-trained knowledge base and fine-tuned capabilities, performs feature extraction and semantic association on the input image patches. The model analyzes visual cues in the image to infer the specific category of the contraband (such as lithium batteries, knives, etc.), and identifies its morphological features (such as shape and size) and attribute information (such as material and state).

[0076] Optionally, the backend multimodal large model is tuned by injecting a low-rank adapter into the attention module of the backend multimodal large model and using the low-rank adapter to fine-tune the model using a pre-built image-text pairing dataset.

[0077] Low-rank adapters are a parameter-efficient fine-tuning technique that replaces some weight updates by injecting a low-rank decomposition matrix (i.e., a low-rank adapter) next to the attention layer of a pre-trained model. This allows for the updating of only a small number of adapter parameters while freezing most of the parameters of the original large model, thereby significantly reducing the computational resources and storage requirements during the fine-tuning process and enabling rapid model adaptation.

[0078] A pre-built image-text pairing dataset contains security inspection image patches and their corresponding contraband description texts. Using this dataset, the backend multimodal large model can learn the mapping relationship between image features and text descriptions. During fine-tuning, the model generates corresponding text descriptions based on the image patch content and optimizes the parameters of the low-rank adapter through a loss function. This makes the backend multimodal large model specifically adapted to security inspection scenarios, improving its accuracy in recognizing the categories, shapes, and attributes of contraband in security inspection images, as well as the quality of description generation. This better serves the collaborative detection process of large and small models.

[0079] Step S103: For the second type of candidate box corresponding to a confidence level greater than or equal to a preset threshold, generate the structured detection information of the second type of candidate box.

[0080] Step S104: Aggregate the natural language text of the image block corresponding to the first type of candidate box and the structured detection information of the second type of candidate box, and output the item security inspection report.

[0081] After the first type of candidate boxes is analyzed by the backend multimodal large model, descriptive natural language text is generated, providing a detailed semantic interpretation of difficult targets. The second type of candidate boxes retains the structured detection data and provides a definite detection result. These two types of information are aggregated in the item security inspection report module to form a complete and multi-dimensional security inspection result. This helps security personnel to fully understand the contents of luggage, improve the ability to identify and understand difficult targets, and maintain the efficiency of regular target detection, thereby improving the accuracy of security inspection work and user experience.

[0082] Through the above steps, at least one security inspection image can be input into a front-end lightweight detection network model. This model is obtained through a hybrid granularity pruning strategy and includes a feature extractor and a neck network. The feature extractor extracts global multi-scale features from the security inspection image to focus on suspicious contraband areas. The neck network performs target perception on these areas, outputting multiple candidate boxes containing image patches and the confidence score of each box. First-class candidate boxes with confidence scores below a preset threshold are selected, and the image patches containing these first-class candidate boxes are input into a back-end multimodal large-scale model. The back-end multimodal large-scale model identifies the category, morphological features, and attribute information of suspicious contraband in the image patches, generating natural language text. For second-class candidate boxes with confidence scores greater than or equal to a preset threshold, structured detection information for these second-class candidate boxes is generated. Finally, the natural language text of the image patches corresponding to the first-class candidate boxes and the structured detection information of the second-class candidate boxes are aggregated to output an item security inspection report. In this embodiment, a feature extractor (using a Transformer architecture) and a neck network can be used to effectively detect small and occluded targets in security inspection images. Simultaneously, a hybrid granularity pruning strategy is employed to reduce the size of the front-end lightweight detection network model, meeting the real-time requirements of edge devices and shortening inference time. Furthermore, this invention utilizes a large-model collaborative mechanism—combining front-end lightweight detection with back-end multimodal large-model analysis—to perform deep semantic analysis and text description generation on low-confidence candidate regions, significantly reducing the false negative rate and improving the interpretability of the detection. This addresses the technical problem in security inspection scenarios where deep learning-based target detection algorithms tend to overlook small target features, resulting in a high false negative rate.

[0083] According to another aspect of the present invention, a method for detecting items in security inspection images is also provided, comprising: inputting at least one acquired X-ray security inspection image into a front-end lightweight detection network model, wherein the front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy, in which a visual backbone network based on a Transformer architecture is used for feature extraction, and the extracted global multi-scale features are input into a convolutional attention unit to focus on the region of suspicious contraband items, multi-scale fusion is performed using a neck network, and a detection head is used to output multiple candidate boxes containing image patches and the confidence score of each candidate box; for confidence scores greater than or equal to... Candidate boxes with a pre-set confidence threshold are marked as high-confidence results. The bounding box coordinates, confidence values, category information, and morphological features of the suspected contraband are retained to obtain structured detection information. For candidate boxes with confidence values ​​lower than the pre-set confidence threshold, they are marked as low-confidence results. The image blocks corresponding to the candidate boxes of low-confidence results are input into the back-end multimodal large model. The back-end multimodal large model identifies the category, morphological features, and attribute information of the suspected contraband in the image blocks and generates natural language text. The natural language text of the image blocks corresponding to low-confidence results and the structured detection information of the image blocks corresponding to high-confidence results are aggregated to output an item security inspection report.

[0084] The following describes in detail another optional implementation method.

[0085] This invention provides a high-precision, lightweight X-ray contraband detection method that can simultaneously meet the real-time requirements of edge devices and the high-precision requirements in complex scenarios. Specifically, it proposes a target detection model based on the SwinTransformer backbone network and CBAM (Convolutional Block Attention Module) to achieve high-precision identification and localization of contraband such as fire-related items, lithium batteries, and sharp objects. The model is optimized through pruning and lightweighting techniques to ensure stable operation on edge devices with an average inference time of ≤500 milliseconds.

[0086] The present invention constructs a "large and small model collaboration" recognition mechanism. The front-end lightweight model is responsible for real-time initial screening and localization, while the back-end multimodal large model performs deep semantic analysis and description generation on low-confidence targets, thereby improving the overall system performance and user experience.

[0087] Figure 2 This is a schematic diagram of an optional security inspection image-based contraband detection system according to an embodiment of the present invention, such as... Figure 2 As shown, the system includes: an input layer, a preprocessing layer, a network model processing layer, and an output layer.

[0088] A single or continuous X-ray security inspection image is input to the input layer. After image preprocessing by the preprocessing layer, it is input to the network model processing layer (including the Swing Transformer module and the CBAM target detection module) for fast inference, and outputs the detection box and confidence score. For high confidence results, the image is output directly. For low confidence or complex regions, the corresponding image block and confidence information are input to the fine-tuned multimodal large model (a multimodal large model with visual-language understanding capabilities) for deep semantic analysis, generating descriptive text, and then outputting the security inspection report.

[0089] The specific implementation steps include:

[0090] Step 1, data preparation.

[0091] Construct a high-quality, annotated X-ray image dataset of no fewer than 10,000 images, including at least 2,000 dual-view images, covering various targets to be identified. The dataset comprises two sources: publicly available datasets and a self-built dataset. The publicly available dataset integrates images from publicly available X-ray security inspection datasets such as SIXray, OPIXray, and HiXray. Categories with insufficient samples in the publicly available datasets are supplemented using the self-built dataset. The self-built dataset consists of actual X-ray images collected at security checkpoints in partner airports. After anonymization, these images are annotated by professionals to supplement categories with insufficient samples in the publicly available datasets (such as fire types and lithium batteries).

[0092] All images are divided into training, validation, and test sets in a fixed ratio (e.g., 7:2:1). Data augmentation is performed on the training set, including random horizontal flipping, color adjustment, and noise addition. At the same time, to meet the fine-tuning requirements of the backend multimodal large model, no less than 10,000 pairs of image-contraband description text are constructed.

[0093] Step 2, front-end model training.

[0094] A Swing transformer + CBAM + FPN + head detector model was constructed, using the AdamW optimizer and cosine annealing for learning rate scheduling. A composite loss function was defined, and the loss was calculated on the validation set after each training epoch to monitor model performance. Early stopping was implemented to save the best model.

[0095] In the lightweight front-end detection network, the Swing Transformer is used as the backbone network, and a CBAM convolutional block attention module is embedded after its output to form the Swing-CBAM feature extractor. The neck network uses FPN for multi-scale fusion and adopts a YOLO-style detection head to output the contraband bounding box and confidence score. The Swing Transformer employs a hierarchical design and a window-shifting self-attention mechanism, which can effectively capture the global contextual information of the image while maintaining sensitivity to local details.

[0096] Based on the multi-scale features extracted by the backbone network, a CBAM attention mechanism is introduced, including channel attention and spatial attention. Channel attention recalibrates the weights of each channel by aggregating global information, emphasizing information-rich feature channels, while spatial attention focuses on key regions by paying attention to their spatial location in the feature map. Integrating the CBAM module into the feature fusion path of the Feature Pyramid Network (FPN) enables the model to adaptively focus on suspicious contraband regions in the image, significantly improving the perception and recognition capabilities for small targets and severely occluded targets.

[0097] Through the above implementation methods, the ability to extract features of contraband in complex backgrounds in X-ray images can be improved by embedding the CBAM attention module in the Swin Transformer backbone network. The hierarchical design and window-shifting self-attention mechanism of the Swin Transformer help capture global contextual information of the image while preserving local details. The CBAM module, through the combination of channel attention and spatial attention, helps the model adaptively focus on suspicious contraband areas, especially for small targets and severely occluded targets, which helps to significantly enhance the saliency and distinguishability of features, thereby improving detection accuracy.

[0098] To address the computational burden of the Swing Transformer, this invention proposes a hybrid pruning method that first global and then local pruning. Using the L1 norm as the criterion, the weights of all Conv2d (two-dimensional convolutional layers) and Linear layers (fully connected layers) within the model are globally ranked by importance, and parameters with lower importance are pruned, achieving global unstructured pruning. Based on global pruning, redundant convolutional channels are structurally removed, achieving local channel-level structured pruning.

[0099] To enable faster and more accurate model convergence, this invention provides a jointly optimized composite loss function, which consists of a weighted average of coordinate loss, IoU loss, confidence loss, and classification loss:

[0100] 1) Coordinate loss defines the regression error of the center point and width and height of the prediction box to ensure accurate positioning.

[0101] 2) IoU loss is used to optimize the intersection-union ratio between the predicted bounding box and the ground truth bounding box, thereby helping to improve the localization quality.

[0102] 3) Confidence loss is the confidence level predicted using a binary cross-entropy correlation model compared to the actual positioning accuracy.

[0103] 4) Classification loss: For the detection of contraband of a specific class, a single-class foreground binary cross-entropy loss is used.

[0104] In this model, the confidence supervision signal for positive samples is set as the intersection-union ratio (IoU) between the predicted bounding box and the ground truth box. In the loss function, the confidence loss adopts the binary cross-entropy loss, and its target value is the dynamically calculated IoU value. This allows the model to learn the presence or absence of targets while also learning precise localization, which can improve the localization accuracy of the detection boxes.

[0105] By using a composite loss function, the misalignment between bounding box regression and classification confidence can be effectively resolved, thereby improving the reliability of model evaluation.

[0106] To meet the real-time requirements of edge devices, a lightweighting method based on L1 norm importance evaluation is used to perform global unstructured pruning on the trained high-precision model. The L1 norm of all Conv2d and Linear layer parameters in the model is calculated, and weights with smaller absolute values ​​are pruned. Through multiple iterative pruning steps, redundant parameters are gradually removed, compressing the model size with minimal loss of accuracy.

[0107] Step 3, lightweight processing.

[0108] Load the trained model and perform global pruning on it; perform local structured pruning to remove redundant channels.

[0109] Step 4: Confidence threshold setting.

[0110] The initial threshold is randomly set within the range of [0.3, 0.9]. After the initial threshold is set, it is dynamically adjusted in steps of 0.05 on the validation dataset. The optimal threshold is selected according to the principle of maximizing F1-score. Experiments have verified that the optimal threshold range is within the range of [0.5, 0.8]. The preferred default solution is 0.65.

[0111] Step 5: Backend multimodal fine-tuning.

[0112] A pre-trained multimodal large model (preferably a multimodal large model with visual-language understanding capabilities) is loaded, and the LoRA (Low-Rank Adaptation) parameter efficient fine-tuning method is used to adapt the model, optimizing it to enable it to describe contraband in X-ray images. Specific implementation methods include: injecting a LoRA adapter into the attention module, fine-tuning using a constructed 10,000-image-text pairing dataset, freezing the base model parameters during training, and updating only the LoRA adapter parameters.

[0113] In the backend multimodal analysis network, a pre-trained multimodal large model is selected. The large model is fine-tuned by constructing an image-text pairing dataset. The low-confidence candidate region images selected by the front-end network are received, and deep inference analysis is performed to output the category, attributes and text description of the prohibited items.

[0114] Step 6, Collaborative Reasoning.

[0115] The input image is fed into the front-end model to obtain the detection box. For regions with a confidence level greater than or equal to a set threshold, the detection box is output directly. If the confidence level is less than the set threshold, the corresponding image is input into the back-end multimodal model, and the back-end model outputs a text description.

[0116] The above-described security image contraband detection solution based on large and small model collaboration addresses the issues of low confidence and lack of description when a single detection model faces challenging samples by utilizing a front-end detection + back-end analysis collaborative approach between large and small models. The lightweight front-end model handles the full throughput and millisecond-level initial screening, directly outputting results with confidence levels above the threshold. For low-confidence candidate regions with confidence levels below the threshold, the image information is input into the large back-end model for analysis. The finely tuned large model can perform deep semantic understanding and reasoning on the low-confidence target images transmitted from the front end and generate text descriptions.

[0117] The above implementation method utilizes an IoU-aware confidence supervision strategy to address the misalignment between bounding box regression and classification confidence. By using the intersection-union ratio (IoU) between the predicted and ground truth bounding boxes as the target value for the confidence loss, the model can learn not only the existence of the target but also the accuracy of target localization, thereby improving the localization quality and evaluation reliability of the detected bounding boxes.

[0118] Through the above implementation methods, a hybrid granularity pruning approach can be used to achieve model lightweighting without significant loss of accuracy. Combining L1-norm-based global unstructured pruning with local structured pruning targeting convolutional channels helps to significantly reduce the number of model parameters and computational cost. Simultaneously, a collaborative identification mechanism between large and small models enhances the system's ability to handle difficult samples. The lightweight front-end model handles the initial screening of the entire sample flow at millisecond levels, directly outputting structured results for high-confidence targets to ensure efficiency. For low-confidence or complex regions, the back-end multimodal large model utilizes its powerful semantic understanding capabilities for deep analysis and text description generation. This collaborative mechanism compensates for the lack of description and inaccurate judgments in low-confidence scenarios by single detection models, improving the overall performance and interpretability of the system.

[0119] The following is a detailed description with reference to another embodiment.

[0120] Example 2

[0121] The item detection device for security inspection images provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above.

[0122] Figure 3 This is a schematic diagram of an optional item detection device for security screening images according to an embodiment of the present invention, such as... Figure 3 As shown, the item detection device for security inspection images may include: a security inspection image input unit 31, a back-end large model recognition unit 32, a high-confidence candidate box information generation unit 33, and an item security inspection report output unit 34.

[0123] The security inspection image input unit 31 is used to input at least one security inspection image into the front-end lightweight detection network model. The front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy. The front-end lightweight detection network model includes a feature extractor and a neck network. The feature extractor is used to extract global multi-scale features of the security inspection image to focus on the suspicious contraband area. The neck network is used to perform target perception on the suspicious contraband area and output multiple candidate boxes containing image patches and the confidence of each candidate box.

[0124] The backend large model recognition unit 32 is used to filter the first type of candidate boxes corresponding to confidence scores below a preset threshold, input the image block of the first type of candidate box into the backend multimodal large model, and the backend multimodal large model recognizes the category, morphological features and attribute information of the suspected contraband in the image block to generate natural language text.

[0125] The high-confidence candidate box information generation unit 33 is used to generate structured detection information for the second type of candidate box corresponding to a confidence level greater than or equal to a preset threshold.

[0126] The item inspection report output unit 34 is used to aggregate the natural language text of the image block corresponding to the first type of candidate box and the structured detection information of the second type of candidate box, and output the item inspection report.

[0127] The aforementioned item detection device for security inspection images can input at least one security inspection image into a front-end lightweight detection network model via the security inspection image input unit 31. This front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy and includes a feature extractor and a neck network. The feature extractor extracts global multi-scale features from the security inspection image to focus on suspicious contraband areas. The neck network performs target perception on these areas, outputting multiple candidate boxes containing image patches and the confidence level of each candidate box. The back-end large model recognition unit 32 then filters items below a certain threshold. The first type of candidate box corresponding to a confidence level of a preset threshold is input into the back-end multimodal large model. The back-end multimodal large model identifies the category, morphological features, and attribute information of the suspected contraband in the image block and generates natural language text. For the second type of candidate box corresponding to a confidence level greater than or equal to the preset threshold, the high-confidence candidate box information generation unit 33 generates the structured detection information of the second type of candidate box. The item security inspection report output unit 34 aggregates the natural language text of the image block corresponding to the first type of candidate box and the structured detection information of the second type of candidate box and outputs the item security inspection report. In this embodiment, a feature extractor (using a Transformer architecture) and a neck network can be used to effectively detect small and occluded targets in security inspection images. Simultaneously, a hybrid granularity pruning strategy is employed to reduce the size of the front-end lightweight detection network model, meeting the real-time requirements of edge devices and shortening inference time. Furthermore, this invention utilizes a large-model collaborative mechanism—combining front-end lightweight detection with back-end multimodal large-model analysis—to perform deep semantic analysis and text description generation on low-confidence candidate regions, significantly reducing the false negative rate and improving the interpretability of the detection. This addresses the technical problem in security inspection scenarios where deep learning-based target detection algorithms tend to overlook small target features, resulting in a high false negative rate.

[0128] Optionally, the feature extractor includes: a visual backbone network based on the Transformer architecture, wherein the visual backbone network introduces a hierarchical feature extraction mechanism and a moving window self-attention mechanism. The hierarchical feature extraction mechanism is used to progressively downsample the security inspection image by stacking multiple stages to generate a global multi-scale feature map. The moving window self-attention mechanism is used to divide the security inspection image into non-overlapping windows, calculate the self-attention mechanism within each window, and offset the window division to allow information interaction between different windows, so as to initially screen suspicious regions of the security inspection image; a convolutional block attention unit, including a channel attention module and a spatial attention module, wherein the channel attention module is used to aggregate global information of the security inspection image to recalibrate the weights of each channel to obtain a feature map, and each channel is responsible for detecting different image features; the spatial attention module is used to focus on the spatial position in the feature map, and combined with the initial screening of suspicious regions of the security inspection image, outputs a spatial attention map containing the suspicious regions.

[0129] Optionally, the neck network includes a feature pyramid network. After the spatial attention map is input into the feature pyramid network, multi-scale feature fusion is performed by the feature fusion path of the feature pyramid network, and the detection head outputs the bounding box coordinates and corresponding confidence scores of all suspected contraband areas.

[0130] Optionally, the item detection device for security inspection images includes, when training the lightweight detection network model at the front end: an image integration unit, which integrates security inspection images from multi-source public datasets and obtains actual images of items passing through security checkpoints collected by collaborating partners through a federated learning framework. After desensitizing the actual images, image annotation is performed to construct a security inspection image dataset containing preset annotation information. The security inspection image dataset contains multiple dual-view images, and each security inspection image contains various types of targets to be identified. A dataset partitioning unit is used to partition the security inspection image dataset into [various categories] according to a fixed ratio. The system comprises training, validation, and test sets. Random data augmentation is applied to the training set, and multiple pairs of security inspection images and text descriptions of prohibited items are constructed. Data augmentation includes horizontal flipping, color adjustment, and noise addition. A model training unit is used to train the lightweight front-end detection network model using the training set after initializing the model architecture. A composite loss function is used for evaluation. After each training epoch, the loss is calculated on the validation set to monitor the performance parameters of the lightweight front-end detection network model. Training stops when the performance parameters meet the requirements, resulting in the trained lightweight front-end detection network model.

[0131] Optionally, the composite loss function includes: a coordinate loss sub-function, used to define the regression error of the predicted candidate box center point and width and height; an intersection-union ratio (IoU) loss sub-function, used to optimize the intersection-union ratio between the predicted candidate box and the ground truth box; a confidence loss sub-function, used to correlate the confidence of the candidate box output by the lightweight detection network model with the actual localization accuracy using binary cross-entropy, with the target confidence value being the intersection-union ratio between the predicted candidate box and the ground truth box; and a classification loss sub-function, which uses single-class foreground binary cross-entropy loss for the detection of contraband of a specific class.

[0132] Optionally, the hybrid granularity pruning strategy includes: using the L1 norm as a criterion, globally ranking the weights of all two-dimensional convolutional layers and fully connected layers in the front-end lightweight detection network model by importance, and pruning weights with importance below a preset importance threshold to complete global unstructured pruning; based on global unstructured pruning, structurally removing redundant convolutional channels to complete local channel-level structured pruning.

[0133] Optionally, the object detection device for security inspection images includes, when adjusting the back-end multimodal large model: a low-rank adapter injection module for injecting a low-rank adapter into the back-end multimodal large model attention module, and using the low-rank adapter to fine-tune the model using a pre-built image-text pairing dataset.

[0134] Optionally, the item detection device for security inspection images further includes: a security inspection image acquisition unit, used to acquire at least one original security inspection image after the item enters the security inspection station or security inspection channel before inputting at least one acquired security inspection image into the front-end lightweight detection network model, wherein the type of security inspection image includes at least: X-ray image; and an image preprocessing unit, used to perform standardized preprocessing on the original security inspection image to generate a security inspection image in a uniform format, wherein the standardized preprocessing includes at least one of the following: image size normalization, noise removal, and contrast enhancement.

[0135] The aforementioned item detection device for security inspection images may also include a processor and a memory. The aforementioned security inspection image input unit 31, back-end large model recognition unit 32, high confidence candidate box information generation unit 33, item security inspection report output unit 34, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0136] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, they can collaboratively achieve contraband identification from security images based on a size model.

[0137] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0138] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform any of the above-described methods for detecting items in security inspection images.

[0139] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the item detection method for security inspection images according to any one of the above embodiments.

[0140] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the item detection method for security inspection images described in various embodiments of this application.

[0141] This application also provides a computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the item detection method for security inspection images described in various embodiments of this application.

[0142] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) that performs an item detection method for security inspection images according to an embodiment of the present invention. Figure 4 As shown, an electronic device may include one or more ( Figure 4 The processor 402 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and the memory 404 for storing data may also be included. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include... Figure 4 The more or fewer components shown, or having the same Figure 4The different configurations shown.

[0143] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0144] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0149] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting items in security inspection images, characterized in that, include: At least one security inspection image is input into a front-end lightweight detection network model, wherein the front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy. The front-end lightweight detection network model includes a feature extractor and a neck network. The feature extractor is used to extract global multi-scale features of the security inspection image to focus on suspicious contraband areas. The neck network is used to perform target perception on the suspicious contraband areas and output multiple candidate boxes containing image patches and the confidence of each candidate box. The first type of candidate box is selected based on the confidence level below a preset threshold. The image block of the first type of candidate box is input into the back-end multimodal large model. The back-end multimodal large model identifies the category, morphological features and attribute information of the suspected contraband in the image block and generates natural language text. For the second type of candidate box corresponding to a confidence level greater than or equal to a preset threshold, generate the structured detection information of the second type of candidate box; The system aggregates the natural language text of the image blocks corresponding to the first type of candidate boxes and the structured detection information of the second type of candidate boxes to output an item security inspection report.

2. The method for detecting items in security inspection images according to claim 1, characterized in that, The feature extractor includes: A visual backbone network based on the Transformer architecture is proposed, wherein the visual backbone network introduces a hierarchical feature extraction mechanism and a moving window self-attention mechanism. The hierarchical feature extraction mechanism is used to progressively downsample the security inspection image by stacking multiple stages to generate a global multi-scale feature map. The moving window self-attention mechanism is used to divide the security inspection image into non-overlapping windows, calculate the self-attention mechanism within each window, and offset the window division to enable information interaction between different windows, so as to initially screen the suspicious areas of the security inspection image. The convolutional block attention unit includes a channel attention module and a spatial attention module. The channel attention module is used to aggregate global information of the security inspection image to recalibrate the weights of each channel to obtain a feature map, and each channel is responsible for detecting different image features. The spatial attention module is used to focus on the spatial position in the feature map, and combined with the preliminary screening of suspicious areas in the security inspection image, outputs a spatial attention map containing the suspicious areas.

3. The method for detecting items in security inspection images according to claim 2, characterized in that, The neck network includes: After the spatial attention map is input into the feature pyramid network, multi-scale feature fusion is performed by the feature fusion path of the feature pyramid network, and the detection head outputs the bounding box coordinates and corresponding confidence scores of all suspected contraband areas.

4. The method for detecting items in security inspection images according to claim 1, characterized in that, Training the aforementioned lightweight front-end detection network model includes: Security inspection images from multiple public datasets are integrated, and actual images of security checks collected at the security check sites of collaborating partners are obtained through a federated learning framework. After the actual images of security checks are desensitized, image annotation processing is performed to construct a security inspection image dataset containing preset annotation information. The security inspection image dataset contains multiple dual-view images, and each security inspection image contains various targets that need to be identified. The security inspection image dataset is divided into a training set, a validation set, and a test set according to a fixed ratio. Random data augmentation processing is performed on the training set, and multiple pairs of text pairs describing security inspection images and prohibited items are constructed. The data augmentation processing includes horizontal flipping, color adjustment, and noise addition. After initializing the architecture of the lightweight front-end detection network model, it is trained using the training set and evaluated using a composite loss function. After each training epoch, the loss is calculated on the validation set, and the performance parameters of the lightweight front-end detection network model are monitored. When the performance parameters meet the requirements, training is stopped, and the trained lightweight front-end detection network model is obtained.

5. The item detection method for security inspection images according to claim 4, characterized in that, The composite loss function includes: The coordinate loss sub-function is used to define the regression error for the predicted candidate box center point and width and height; The Intersection over Union (IoU) loss function is used to optimize the intersection-over-union ratio between predicted candidate boxes and ground truth labeled boxes. The confidence loss function is used to correlate the confidence of the candidate boxes output by the front-end lightweight detection network model with the actual localization accuracy using binary cross-entropy. The target confidence value is the intersection-union ratio of the predicted candidate boxes and the ground truth boxes. The classification loss function employs a single-class foreground binary cross-entropy loss for detecting contraband of a specific class.

6. The method for detecting items in security inspection images according to claim 1, characterized in that, The hybrid granularity pruning strategy includes: Using the L1 norm as the criterion, the weights of all two-dimensional convolutional layers and fully connected layers in the front-end lightweight detection network model are globally ranked by importance, and weights with importance lower than a preset importance threshold are pruned to complete global unstructured pruning. Based on global unstructured pruning, redundant convolutional channels are structurally removed to complete local channel-level structured pruning.

7. The method for detecting items in security inspection images according to claim 1, characterized in that, The backend multimodal large model is adjusted in the following way: A low-rank adapter is injected into the backend multimodal large model attention module, and the model is fine-tuned using a pre-built image-text pairing dataset with the low-rank adapter.

8. The method for detecting items in security inspection images according to claim 1, characterized in that, Before inputting at least one acquired security inspection image into the front-end lightweight detection network model, the following steps are also included: After an item enters the security checkpoint or security check lane, at least one original security image is captured, wherein the type of the security image includes at least: an X-ray image; The original security inspection image is subjected to standardized preprocessing to generate a security inspection image in a uniform format. The standardized preprocessing includes at least one of the following: image size normalization, noise removal, and contrast enhancement.

9. A method for detecting items in security inspection images, characterized in that, include: At least one X-ray security inspection image is input into a front-end lightweight detection network model, wherein the front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy. In the front-end lightweight detection network, a visual backbone network based on the Transformer architecture is used for feature extraction, and the extracted global multi-scale features are input into a convolutional attention unit to focus on the suspicious contraband area. A neck network is used for multi-scale fusion, and the detection head outputs multiple candidate boxes containing image patches and the confidence of each candidate box. For candidate boxes with a confidence value greater than or equal to the preset confidence threshold, they are marked as high-confidence results. The bounding box coordinates, confidence value, category information and morphological features of the suspected contraband are retained to obtain structured detection information. For candidate boxes with confidence scores less than a preset confidence threshold, they are marked as low-confidence results. The image blocks corresponding to the candidate boxes of the low-confidence results are then input into the back-end multimodal large model. The back-end multimodal large model identifies the category, morphological features, and attribute information of the suspected contraband in the image blocks and generates natural language text. The system aggregates the natural language text of the image patches corresponding to the low-confidence results and the structured detection information of the image patches corresponding to the high-confidence results, and outputs an item security inspection report.

10. An item detection device for security inspection images, characterized in that, include: A security inspection image input unit is used to input at least one acquired security inspection image into a front-end lightweight detection network model. The front-end lightweight detection network model is obtained through a hybrid granularity pruning strategy. The front-end lightweight detection network model includes a feature extractor and a neck network. The feature extractor is used to extract global multi-scale features of the security inspection image to focus on suspicious contraband areas. The neck network is used to perform target perception on the suspicious contraband areas and output multiple candidate boxes containing image patches and the confidence of each candidate box. The backend large model recognition unit is used to filter the first type of candidate boxes corresponding to confidence scores below a preset threshold, input the image block of the first type of candidate box into the backend multimodal large model, and the backend multimodal large model recognizes the category, morphological features and attribute information of the suspected contraband in the image block to generate natural language text; The high-confidence candidate box information generation unit is used to generate structured detection information for the second type of candidate box corresponding to a confidence level greater than or equal to a preset threshold. The item security inspection report output unit is used to aggregate the natural language text of the image blocks corresponding to the first type of candidate boxes and the structured detection information of the second type of candidate boxes, and output the item security inspection report.

Images