Method and system for detecting customs prohibited articles based on multi-modal adaptive fusion

Abstract

The application discloses a customs prohibited article detection method and system based on multi-modal adaptive fusion, comprising: obtaining a customs clearance image-text pair, and extracting original features including multi-scale features of the customs clearance image; shallow fusion features and deep fusion features are generated after weighting fusion of shallow features and deep features respectively; a pooling feature is obtained after multi-scale pooling operation of the deep fusion features; enhanced features are generated by splicing the shallow fusion features, the deep fusion features and the pooling feature; a visual feature is obtained after integration of the enhanced features and the original features; the visual feature and a text feature are aligned in mode, and a two-stage training is performed on a detection model, so that a prohibited article detection result is obtained. An adaptive dynamic channel fusion module is designed to integrate shallow details and deep semantics from multi-layer hidden states, optimize feature expression through adaptive weights, effectively fuse low-level details and deep semantics, and improve feature resolution capability in complex scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of customs contraband detection, and in particular to a customs contraband detection method and system based on multi-modal adaptive fusion. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the increasing demand for customs supervision, multi-modal data (such as text, images, videos, etc.) are gradually attracting attention in the application of goods detection, contraband identification and customs clearance efficiency improvement. In the prior art, the processing and feature extraction methods of multi-modal information have been applied to the customs supervision field to a certain extent, but there are still many deficiencies, which limit the performance of the system in complex scenarios and the actual application effect.

[0004] In recent years, multi-modal large models have realized cross-modal feature fusion through large-scale pre-training and fine-tuning based on pre-trained models, which can more comprehensively understand the characteristics of goods and improve the supervision ability. However, the existing technology has limitations in multi-level feature fusion. Traditional visual encoders usually only extract deep semantic features, ignoring low-level detailed information, which is particularly disadvantageous in customs contraband detection. For densely arranged goods or hidden contraband, the lack of low-level features makes it difficult for the model to capture subtle differences, resulting in a decline in recognition ability; and the existing methods often use simple channel concatenation when fusing multi-level features, which is complex in structure and low in efficiency, and cannot effectively balance performance and computational cost.

[0005] In the aspect of visual feature extraction, the existing technology generally relies on high-resolution image input or a large number of visual markers to improve precision, but this approach significantly increases computational complexity, especially when processing multiple images or video data, the feature sequence length is too long, resulting in high consumption of computing resources and a significant decrease in inference speed. For example, the feature extraction method based on traditional Transformer model, the computational complexity of its attention mechanism is proportional to the square of the sequence length, making it difficult to meet the real-time needs of customs high-throughput detection. In order to pursue performance improvement, increasing the resolution or the number of markers often requires complex preprocessing steps, further increasing the operational complexity and system implementation cost.

[0006] The prior art lacks effective optimization strategies in dealing with the sparsity and imbalance of customs multi-modal data. In the customs multi-modal data, the sample quantity and quality of images and texts differ greatly. For example, the amount of image data can be much larger than the amount of text description, and the distribution of prohibited item categories is unbalanced (e.g., there is more data of common prohibited items and insufficient data of rare prohibited items), which causes the contribution of certain modalities or categories to be amplified or weakened during model training, affecting the comprehensiveness and accuracy of the analysis results; if the model relies too much on visual information or common category data, it may overlook the key clues in the text and the characteristics of rare prohibited items, thereby reducing the overall detection performance. SUMMARY

[0007] To solve the above problems, the present application provides a customs prohibited item detection method and system based on multi-modal adaptive fusion, which designs an adaptive dynamic channel fusion module to integrate shallow details and deep semantics from multiple hidden states, optimizes feature expression through adaptive weights, effectively fuses low-level details and deep semantics, and improves feature discrimination ability in complex scenarios.

[0008] To achieve the above purpose, the present application adopts the following technical solutions:

[0009] In a first aspect, the present application provides a customs prohibited item detection method based on multi-modal adaptive fusion, comprising:

[0010] Obtaining a customs clearance image-text pair and extracting the original features of the customs clearance image including multi-scale features;

[0011] Taking the first N layers of features in the multi-scale features as shallow features and the remaining layers of features as deep features, respectively weighting and fusing the shallow features and the deep features to generate shallow fusion features and deep fusion features, obtaining a pooling feature after multi-scale pooling operation on the deep fusion features, concatenating the shallow fusion features, the deep fusion features and the pooling feature to generate an enhanced feature, and integrating the enhanced feature and the original feature to obtain a visual feature;

[0012] Aligning the visual feature and the text feature in modalities, training the detection model in two stages based on the obtained aligned multi-modal features, and obtaining a prohibited item detection result by using the trained detection model on the customs clearance image to be tested.

[0013] As an optional implementation, after obtaining the customs clearance image, the customs clearance image is divided into a×b cropped blocks, and the total number of visual Tokens is L=(a×b+1)×T; when the total number of visual Tokens L exceeds the set Token quantity threshold τ, the Token quantity is adjusted, and the optimal spatial configuration is selected;

[0014]

[0015] wherein T new is the adjusted Token number of each cropped block, T is the original Token number of each cropped block; a, b represent the number of divided columns and rows; a x b + 1 is the total number of cropped blocks plus the original image, which is used to calculate the total number of cropped units.

[0016] As an optional implementation, the weight of each layer feature is calculated for weighted fusion;

[0017]

[0018] After normalization, the shallow adaptive weight and the deep adaptive weight are generated respectively:

[0019]

[0020] wherein W s i is the weight of the i-th shallow feature; is the weight of the j-th deep feature; S i [:,l,d] is the value of the i-th shallow feature at the l-th position and the d-th dimension; D j [:,l,d] is the value of the j-th deep feature at the l-th position and the d-th dimension; L is the sequence length, and D is the hidden dimension; N s is the number of shallow layers, and N d is the number of deep layers.

[0021] As an optional implementation, the process of performing a multi-scale pooling operation on the deep fusion feature includes:

[0022] performing one-dimensional average pooling on the deep fusion feature F d , and replacing the dimension to change the shape to [B, D, L];

[0023] applying a pooling operation to output a shape of [B, D, L'], wherein

[0024] adjusting the sequence length of the deep pooling feature to L through a linear interpolation operation , and adjusting the dimension order to [B, L, D]; B is the batch size, L ′ and L are the sequence lengths, and D is the hidden dimension.

[0025] As an alternative embodiment, for the visual feature representation, a bilateral soft matching strategy is introduced, the cosine similarity is used to measure the Token similarity, the Token pair with the highest similarity is matched and fused to generate a new Token; specifically including: dividing all Tokens into two sets A and B, for each Token in set A, using cosine similarity to find the most similar Token in set B, and connecting with an edge; after matching, the original Token is fused by weighted average to generate a new Token.

[0026] As an alternative embodiment, the paired image-text pair is combined with the corresponding instruction template to construct a visual instruction fine-tuning dataset for model fine-tuning; and the visual features are mapped to a semantic space consistent with the text features through a double-layer mapping layer to realize modality alignment; then the two-stage training process includes: the first stage is a cross-modal mapping optimization stage, the parameters of the visual encoder and the language encoder are frozen, the double-layer mapping layer is trained to optimize the mapping relationship between the visual features and the text features, and preliminary alignment between modalities is realized; the second stage is a visual instruction fine-tuning stage, the parameters of the visual encoder and the language encoder are unfrozen, and the double-layer mapping layer is trained together to train the overall model.

[0027] As an alternative embodiment, for a sequence of length L, the probability p(X a |X a ,X v ,X q ) is defined as:

[0028]

[0029] Where X v is the visual feature after modality alignment; X q is the text feature of the instruction template; X q,＜i is all Tokens of the instruction template, X a,<i is the first i-1 Tokens in the answer, respectively representing the instruction and answer context processed by the language encoder when predicting the current answer Token x i ;

[0030] is represented as the language encoder predicting each Token x i of the answer one by one in an autoregressive manner, at each prediction, the visual feature X v , the instruction Token X q,＜i and the generated answer Token X a,＜i are referenced to calculate the probability p(x i |X v ,X q,＜i ,X a,＜i), and finally multiply the probabilities of all Tokens to obtain the probability of the entire answer; the finally generated result includes the name of the prohibited article and the position of the bounding box.

[0031] In a second aspect, the present application provides a customs prohibited article detection system based on multi-modal adaptive fusion, comprising:

[0032] A feature extraction module is configured to obtain a customs clearance image-text pair and extract original features of the customs clearance image, including multi-scale features;

[0033] An adaptive fusion module is configured to take the first N layers of features in the multi-scale features as shallow features and the remaining layers of features as deep features, respectively, to generate shallow fusion features and deep fusion features after weighted fusion of the shallow features and the deep features, to obtain a pooling feature after multi-scale pooling operation on the deep fusion features, to splice the shallow fusion features, the deep fusion features and the pooling feature to generate an enhanced feature, and to integrate the enhanced feature and the original feature to obtain a visual feature;

[0034] A training module is configured to align the visual feature with a text feature in modalities, to perform two-stage training on a detection model based on the obtained aligned multi-modal feature, and to obtain a prohibited article detection result for a to-be-tested customs clearance image by using the trained detection model.

[0035] In a third aspect, the present application provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein when the computer instructions are run by the processor, the method of the first aspect is completed.

[0036] In a fourth aspect, the present application provides a computer readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method of the first aspect is completed.

[0037] In a fifth aspect, the present application provides a computer program product comprising a computer program, wherein when the computer program is executed by a processor, the method of the first aspect is completed.

[0038] Compared with the prior art, the present application has the following beneficial effects:

[0039] The prior art fails to fully utilize the potential of the pre-trained visual encoder in multi-level feature fusion, ignores the extraction of low-level detail information, and thus limits the recognition ability in complex scenes. Therefore, in view of the problem that the prior art ignores low-level details in multi-level feature fusion and is difficult to recognize dense or hidden contraband, the present application designs an adaptive dynamic channel fusion module to integrate shallow details and deep semantics from multiple hidden states, optimizes feature expression through adaptive weights, effectively fuses low-level details and deep semantics, and improves feature discrimination ability in complex scenes. At the same time, the balance and utilization of customs multi-modal data are optimized, the problems of sparsity and imbalance are solved, the collaborative contribution of different modalities and different data types is improved, and the comprehensiveness and accuracy of the analysis results are ensured.

[0040] Compared with the prior art which relies on fixed high-resolution input or a large number of visual markers, resulting in high computational complexity and insufficient inference speed to meet real-time requirements. The present application introduces the AnyRes strategy, dynamically adjusts the configuration according to the resolution of the input customs contraband visual data, so that the model can adaptively receive the original resolution of the picture itself, support adaptive resolution, and thus when inputting image and video data, the details of local visual features and the integrity of global visual information are considered, the model performance is comprehensively improved with little increase in model size, the processing capability of X-ray images of any resolution is optimized, the efficiency of feature extraction is ensured, and the problem of insufficient adaptability of feature extraction caused by the dependence of traditional visual encoder on fixed high-resolution input is solved.

[0041] The present application introduces a bilateral soft matching fusion strategy, effectively reduces the sequence length by matching and fusing similar Token pairs, reduces resource consumption in the inference process, while retaining key information, thereby optimizing the fusion structure, reducing redundant calculation, and ensuring that the system can realize real-time response in a high-throughput detection scenario, improving the balance between system efficiency and computing cost. In the case of low impact on model performance, the computational complexity is greatly reduced and the real-time performance is improved, while solving the problem of excessive computational burden caused by long feature sequences, optimizing the inference speed to meet the real-time needs of customs high-throughput detection.

[0042] In view of the problem that the prior art has insufficient correlation between modalities in the customs contraband detection task, resulting in low feature fusion efficiency and poor semantic consistency. Through a two-stage training method to optimize the alignment of inter-modal features, the visual encoder and the language encoder are frozen in the first stage, only the double-layer mapping layer is trained, and the visual features are accurately mapped to the embedding space of the language encoder to improve the semantic consistency of the visual features and the text features. In the second stage, all parameters are unfrozen for overall fine-tuning, and the detection accuracy of the model in the customs multi-modal task is improved through visual instruction optimization.

[0043] Advantages of the additional aspects of the application will become apparent in the following description. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0045] Figure 1 A flow chart of a customs prohibited article detection method based on multi-modal adaptive fusion is provided for the embodiment 1 of the present application;

[0046] Figure 2 An adaptive dynamic channel fusion schematic diagram is provided for the embodiment 1 of the present application;

[0047] Figure 3 A first stage training flow chart is provided for the embodiment 1 of the present application;

[0048] Figure 4 A second stage training flow chart is provided for the embodiment 1 of the present application. DETAILED DESCRIPTION

[0049] The present application will be further described below in conjunction with the drawings and embodiments.

[0050] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0051] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that the terms "comprise" and "include" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0052] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0053] Embodiment 1

[0054] The embodiment provides a customs prohibited article detection method based on multi-modal adaptive fusion, comprising:

[0055] obtaining a customs clearance image-text pair, and extracting original features of the customs clearance image, the original features comprising multi-scale features;

[0056] taking the first N-layer features in the multi-scale features as shallow features and the remaining layer features as deep features, respectively weighting and fusing the shallow features and the deep features to generate shallow fusion features and deep fusion features, performing multi-scale pooling operation on the deep fusion features to obtain pooled features, splicing the shallow fusion features, the deep fusion features and the pooled features to generate enhanced features, and integrating the enhanced features and the original features to obtain visual features;

[0057] aligning the visual features with text features, performing two-stage training on a detection model based on the obtained aligned multi-modal features, and obtaining a prohibited article detection result by using the trained detection model on a to-be-detected customs clearance image.

[0058] The embodiment will be described in detail below. Figure 1 The embodiment will be described in detail below.

[0059] Step 1: Obtain a customs clearance image-text pair.

[0060] Specifically,

[0061] Firstly, original multi-modal data related to prohibited article detection is collected from a customs clearance supervision system, including customs clearance X-ray images and corresponding labeled texts.

[0062] The X-ray images cover multiple types of goods (such as non-metallic lighters, controlled knives, liquid containers, etc.), and the labeled texts include image IDs (such as “009004.jpg”), prohibited article labels (such as “straight knife”), and prohibited article position coordinates [x1, y1, x2, y2] (such as “[711 523 800 598]”), (x1, y1) being the upper left corner position of the prohibited article, and (x2, y2) being the lower right corner position of the prohibited article, providing diversified input basis for subsequent multi-modal feature extraction.

[0063] Secondly, the customs clearance X-ray images are divided into a global labeling group (labeling object name and bounding box) and a local cropping group (cropping prohibited article region), and the class sample distribution is adjusted through oversampling and undersampling to enhance spatial information and object feature learning.

[0064] Specifically,

[0065] The local cropping group crops the contraband from the original image according to the bounding box, highlights the object details, and facilitates the model to better capture the information features of the object in the visual feature extraction, thereby providing high-quality input for subsequent semantic space mapping.

[0066] The global annotation group generates a bounding box at the position corresponding to the X-ray image according to the position coordinate information of the contraband in the annotation text, and annotates the corresponding object name, so that the model can better learn global information and spatial information in the visual instruction optimization training.

[0067] By analyzing the sample distribution of the contraband categories in the X-ray image, oversampling is used for categories whose number is less than the target threshold, and new samples are generated through image enhancement methods (horizontal flip, vertical flip and rotation); undersampling is used for categories whose number exceeds the target threshold, and samples with higher resolution are retained to the target number to balance the sample proportion of each category and avoid model bias towards the majority class.

[0068] Finally, the X-ray images of the global annotation group and the local cropping group are paired with the corresponding descriptive text extracted from the annotation text to form image-text pairs, ensuring that each image corresponds to its related text content.

[0069] Step 2: According to the requirements of the customs contraband detection task, design the instruction template, which includes user instructions (such as "identify the contraband in the image and classify" and "provide the bounding box position") and expected model responses (such as "detect a knife, position [x, y, w, h]"), to support task-oriented training;

[0070] Combine the paired image-text pairs with the corresponding instruction templates to construct a visual instruction optimization dataset for model fine-tuning; this dataset contains image ID, image file path and dialogue content, which includes multiple rounds of interaction between user instructions and model responses; for example, the user requests "identify the contraband and provide the name and position", and the model returns "find a straight knife, bounding box position [609, 506, 790, 549]" and so on.

[0071] Step 3: Preprocess the visual instruction optimization dataset, including data balancing, data normalization and data cleaning; among them, data normalization converts the image to a tensor and normalizes the pixel value of the image from the range [0, 255] to control the pixel value within [0, 1] to unify the numerical scale; data cleaning eliminates fuzzy images, incomplete pairs or invalid texts to ensure the integrity and applicability of the data.

[0072] Step four: introduce the AnyRes strategy, dynamically adjust the configuration according to the resolution, crop the customs clearance image, process single image, multiple images and video data; then extract features through the visual encoder (SigLIP) to generate original visual features.

[0073] Specifically includes:

[0074] The AnyRes strategy is introduced to dynamically adjust the configuration according to the resolution of the input customs visual data, process single image, multiple images and video data; analyze the maximum number of tokens generated by the AnyRes strategy for different input single images, multiple images and video scenes, so that the model can dynamically adapt to the resolution and type of input image.

[0075] The AnyRes strategy divides the customs clearance image into a×b crop blocks according to the configuration (a, b), and the total number of visual tokens L=(a×b+1)×T; when the total number of visual tokens L exceeds the threshold τ, the number of tokens is adjusted using bilinear interpolation, and the optimal configuration is selected to adapt to different resolutions and aspect ratios:

[0076]

[0077] Where, T new is the adjusted token number of each crop block, T is the original token number of each crop block; a and b are the spatial configuration of the AnyRes strategy, representing the number of columns and rows of division respectively; a×b+1 is the total number of crop blocks plus the base image, which calculates the total number of crop units; τ is the preset token number threshold, which is used to control the total number of visual tokens.

[0078] The original token number of the crop block is the original token number of the image after processing, which is calculated according to the resolution of the input image. Taking a single image as an example, if the input is an image with a resolution of 384×384, the image is divided into patches with a size of 14×14 according to the image division strategy of the visual encoder, then the original token number=(384 / 14)×(384 / 14)=729; according to the AnyRes strategy, the number of tokens generated by the image is (1+9)×729=7290<=7290, so it does not exceed the token number threshold set for single image, so T new is the original 7290. If the resolution of the input image is greater than the threshold, it is adjusted to be less than the threshold 7290 to control the number of tokens.

[0079] For single image, set a large spatial resolution, generate long sequence features by cropping and adjusting the global thumbnail, assign up to 729 visual tokens and each image can have up to 729x(1+9) = 7290 tokens, and obtain local and global visual features through AnyRes pooling.

[0080] For multiple images, each image is scaled to 384x384 pixels, padded to maintain the aspect ratio, input into the visual encoder and remove the padding, and the training data contains up to 12 images, totaling up to 12x729 = 8748 tokens. Avoid high-resolution multi-block cropping to save computing resources.

[0081] For video, each frame is adjusted to the base resolution and then passed through the visual encoder to generate feature maps. A 2x2 bilinear interpolation is used to reduce the number of tokens to 196. A maximum of 32 frames are sampled, for a total of up to 196x32 = 6272 tokens. This balances the number of frames and computational cost.

[0082] In this embodiment, the subsequent bilateral soft matching strategy and probability calculation are based on tokens. The reason is that tokens are the basic unit of multi-modal large models. In this model, both text and images (videos) are uniformly encoded into tokens as the basic unit of input and intermediate results.

[0083] 1. The AnyRes strategy controls the number of tokens. On the one hand, it dynamically adapts the resolution and type of input images, so that input images are not uniformly adjusted to a fixed resolution like traditional models (e.g., input images are adjusted to 224x224). On the other hand, it saves computing resources, as transformers architecture has limitations on input length.

[0084] 2. The subsequent bilateral soft matching strategy reduces the computational cost caused by adci while preserving most of the key information. Therefore, some merging operations need to be performed on tokens.

[0085] 3. Probability calculation is because the model is based on transformers architecture, which is a self-recursive generation model. Therefore, it is based on tokens to predict the next most probable token.

[0086] Subsequently, the visual encoder (SigLIP) cuts the input customs clearance image into fixed-size image patches, converts each image patch into a vector representation through a linear embedding layer while adding a position embedding, and forms an image patch sequence; the image patch sequence is input into a visual backbone network composed of multiple layers of Transformers, and is processed layer by layer through a multi-head self-attention mechanism, and the visual representation is updated at each layer to extract progressively richer image semantic features, i.e., the original image feature sequence F img .

[0087] Among them, the shallow features are usually derived from the first thirteen layers of the network, have a high spatial resolution and strong local structure expression ability, and are suitable for capturing low-level visual information such as edges and textures; the deep features come from the last thirteen layers of the network, have stronger semantic abstraction ability, and can model the context relationship and global dependency structure between targets.

[0088] Generally speaking, the visual encoder will use deep features for the final output. To further mine the semantic information at different levels, shallow features and deep features are extracted from the output of all hidden layers of the visual encoder and input into the ADCI (Adaptive Dynamic Channel Integration) module. The generated enhanced feature sequence and the original image feature sequence are integrated in the channel dimension to realize multi-scale and multi-level feature fusion and enhancement, and generate a more rich and structured high-dimensional visual feature sequence.

[0089] Step five: as shown in Figure 2 , an innovative ADCI (Adaptive Dynamic Channel Integration) module is proposed to extract shallow features and deep features from the hidden layers of the visual encoder.

[0090] Specifically, it includes:

[0091] In order to more fully exert the ability of the visual encoder in visual information extraction, the ADCI module is proposed to optimize the fusion effect of visual information by integrating the multi-level visual features of the visual encoder.

[0092] Through the dense connection strategy and adaptive feature fusion technology, the multi-layer hidden layers H = {H0, H1, …, H N-1} of the visual encoder (SigLIP) (each layer has a shape of [B, L ′ , D]) and the original image feature sequence F img (shape [B, L, D]) are taken as inputs, where B is the batch size, L ′and L is the sequence length, D is the hidden dimension, and N = 26. In the ADCI module, shallow features and deep features are extracted from the hidden layer H.

[0093] The operation steps of extracting shallow features are: shallow features S i (i = 0, …, N s -1), where N s = 13 is the number of shallow layers. All tokens (S i = H i [:, :]) are retained for each layer of hidden layer H i , without performing any slicing operation. Then S i is stacked according to the shape of [N s , B, L, D].

[0094] The operation steps of extracting deep features are: deep features D j (j = N s , …, N-1), where N d = N-N s = 13 is the number of deep layers. All tokens (D j = H j [:, :]) are retained for each layer of hidden layer H j , without performing any slicing operation. Then D j is stacked according to the shape of [N d , B, L, D].

[0095] Step six: adaptive weights are calculated based on the L2 norm of the features in ADCI and weighted fusion to generate shallow fusion features and deep fusion features.

[0096] Specifically, it includes:

[0097] After extracting the shallow features S and the deep features D from the hidden layer H, the L2 norm of each layer is calculated for S and D, that is, as the weight of each layer:

[0098]

[0099]

[0100] wherein, is the weight of the i-th shallow feature; is the weight of the j-th deep feature; S i [:, l, d] is the value of the i-th shallow feature at the l-th position and the d-th dimension; D j [:, l, d] is the value of the j-th deep feature at the l-th position and the d-th dimension; L is the sequence length, and D is the hidden dimension.

[0101] Then generate shallow adaptive weights by Softmax normalization and deep adaptive weights :

[0102]

[0103]

[0104] where the weight shape is [N s ,B,1,1] and [N d ,B,1,1], dynamically reflecting the importance of each layer of features.

[0105] Finally, the shallow fusion features and deep fusion features are generated by weighting fusion using these weights, and the output shape is [B, L ′ ,D].

[0106] Step seven: in ADCI, a multi-scale pooling operation is additionally applied to the deep fusion features to generate deep pooling features.

[0107] Specifically, it includes:

[0108] First, one-dimensional average pooling is performed on F d , and the replacement dimension is The shape becomes [B, D, L].

[0109] Then, the pooling operation is applied, and the output shape is [B, D, L ′ ], where k is the size of the pooling window, indicating that in each pooling operation, the window covers k consecutive elements and takes the average of these elements; s is the step of the pooling, indicating that the pooling window slides s elements forward each time.

[0110] Next, the linear interpolation operation is performed to adjust the sequence length of the deep pooling features to L, and finally the dimension order is adjusted to [B, L, D]; size specifies that the target sequence length is the original L, that is, the length of the interpolated features should be L; mode specifies that the interpolation method is linear interpolation, which fills in the missing values by calculating the linear relationship between the adjacent two points, so that the feature sequence length is restored to L.

[0111] Step eight: in ADCI, the shallow fusion features, deep fusion features and deep pooling features are concatenated in the channel dimension to generate an enhanced feature sequence, with a shape of [B, L, 3D];

[0112] The formula for concatenating features is as follows:

[0113]

[0114] where F s is the shallow fusion feature; F d is the deep fusion feature; is the deep pooling feature; dim = -1 means concatenation along the last dimension.

[0115] Step nine: integrate the enhanced feature sequence and the original image feature sequence along the channel dimension in the ADCI to output the final high-dimensional visual feature sequence with the shape of [B, L, 4D];

[0116] F output = Concat(F img , F ADCI , dim = -1) (7);

[0117] where F img is the original image feature sequence; F ADCI is the enhanced feature sequence after the ADCI module; dim = -1 means concatenation along the last dimension.

[0118] Step ten: in order to ensure the inference speed of the model, a bilateral soft matching strategy is introduced to reduce the length of the high-dimensional visual feature sequence. The cosine similarity is used to measure the Token similarity, and the Token pair with the highest similarity is matched and fused to generate a new Token, so as to reduce the length of the feature sequence while retaining important information.

[0119] Specifically, it includes:

[0120] Divide all the Tokens input into the input Token fusion module into two sets A and B; then for each Token in set A, find the most similar Token in set B and draw an edge to connect them. This matching process is based on cosine similarity measurement, and each matching pair has a similarity weight, reflecting the association strength between Tokens.

[0121] After the matching is completed, the features of multiple original Tokens are fused by weighted average to generate a new Token, which reduces the number of Tokens while making each Token contain more information. The merged Token information can still be propagated in the visual encoder.

[0122] When Token Merging is performed between Attention and Multi-Layer Perceptron, the Token information fused can be propagated to other Tokens through the attention mechanism. The Token fusion algorithm can iteratively perform the above steps in each layer of the visual encoder, gradually reducing the number of Tokens, and further reducing the computational complexity with each layer reduction.

[0123] Step eleven: projecting the visual features into the text semantic space through a double-layer mapping layer (MLP) to realize modal alignment, and generating a prediction result related to the customs task based on the aligned multi-modal features by the language encoder (LLM).

[0124] Specifically includes:

[0125] The double-layer mapping layer (MLP) maps the visual feature sequence into a semantic space consistent with the text features to realize deep alignment between modalities; wherein the text features are formed by the Tokenizer tokenizer generating text Tokens and the embedding layer of the language encoder (LLM) encoding.

[0126] The language encoder (LLM) receives the visual features and text features aligned by the double-layer mapping layer (MLP), and generates a structured result related to the customs prohibited goods detection based on the multi-modal joint input.

[0127] For a sequence of length L, the probability p(X a |X a ,X v ,X q ) is defined as follows:

[0128]

[0129] Where X v is the modal aligned visual feature, generated by the visual encoder and the double-layer mapping layer; X q is the text feature of the instruction template, formed by the Token sequence X q,<i of the instruction template, encoded by the embedding layer and the Transformers layer of the language encoder (LLM); X q,＜i is the Token sequence of the instruction template, X a,＜i is the first i-1 Tokens in the answer, respectively representing the instruction and answer context processed by the language encoder (LLM) when predicting the current answer Token x i .

[0130] Equation (8) represents that the language encoder (LLM) predicts each Token x i of the answer in an autoregressive manner, and each time the prediction is made, the visual feature X vToken sequence X of the instruction template q,＜i and the generated answer Token X a,＜i , calculate the probability p(x i |X v ,X q,＜i ,X a,＜i ), and finally multiply the probabilities of all Tokens to obtain the probability of the entire answer.

[0131] The final generated result includes the name of the prohibited article, the boundary box position (such as [x1, y1, x2, y2] coordinates), and the format is a text report.

[0132] Step twelve: The detection model training adopts a two-stage training method.

[0133] Specifically, it includes:

[0134] As shown in Figure 3 , the first stage is a cross-modal mapping optimization stage, using an open-source pre-training dataset. In this stage, the parameters of the visual encoder and the language encoder are frozen, and only the double-layer mapping layer is trained to optimize the mapping relationship between visual features and text features, achieving preliminary alignment between modalities.

[0135] The cross-entropy loss function is used, and the AdamW optimizer is used to update the parameters. The learning rate is set to η1=1e-3, the gradient accumulation step is 2, the cosine learning rate scheduling is used (the warm-up ratio is 0.03), 3 rounds of training are performed, a checkpoint is saved every 1000 steps, and the maximum input sequence length of the model is 8192 to improve the modal alignment capability.

[0136] As shown in Figure 4 , the second stage is a visual instruction optimization stage. In this stage, the parameters of the visual encoder and the language encoder are unfrozen, and the whole model training is performed together with the double-layer mapping layer to further improve the performance of the model on specific visual tasks.

[0137] The language encoder learning rate η2=1×10 -5 is set, the visual encoder learning rate η vision =2×10 -6 (attention, the language encoder learning rate is set to five times the visual encoder learning rate), the batch size is 1, the gradient accumulation step is 2, the cosine learning rate scheduling is used (the warm-up ratio is 0.03), 1 round of training is performed, a checkpoint is saved every 1000 steps, the maximum input sequence length of the model is 32768, and the gradient checkpoint technology is used to optimize memory usage, further improving the performance of the model on specific visual tasks such as customs prohibited articles detection.

[0138] The customs prohibited article detection method based on multi-modal adaptive fusion provided in this embodiment solves the problem that the prior art ignores low-level details in multi-level feature fusion and is difficult to identify dense or hidden prohibited articles, and designs an ADCI module to integrate shallow details and deep semantics from multi-level hidden states, and optimizes feature expression through adaptive weights, thereby improving feature resolution capability in complex scenes.

[0139] Compared with the prior art which relies on fixed high-resolution input or a large number of visual markers, resulting in high computational complexity and insufficient inference speed to meet real-time requirements. By introducing the AnyRes strategy, the resolution of the input customs prohibited article visual data is dynamically adjusted and configured, so that the model can adaptively receive the original resolution of the picture itself, thereby balancing the details of local visual features and the integrity of global visual information when inputting image and video data, and comprehensively improving the model performance with little increase in model size; At the same time, the Token fusion strategy is introduced, and the redundant Token is clustered and fused using bilateral soft matching, which greatly reduces the inference calculation amount of the model and improves the system processing efficiency with little effect on the performance of the model.

[0140] The prior art has the problem of low feature fusion efficiency and poor semantic consistency in the task of customs prohibited article detection due to insufficient correlation between modalities. Through a two-stage training method to optimize the alignment of features between modalities, in the first stage, the visual encoder and the language encoder are frozen, and only the double-layer mapping layer is trained to accurately map the visual features to the embedding space of the language encoder, so as to improve the semantic consistency of the visual features and the text features. In the second stage, all parameters are unfrozen for overall fine-tuning, and the detection accuracy of the model in the customs multi-modal task is improved through visual instruction optimization.

[0141] It should be noted that all data acquisition is based on compliance with laws and regulations and user consent, and legal application of data.

[0142] Embodiment 2

[0143] The embodiment provides a customs prohibited article detection system based on multi-modal adaptive fusion, comprising:

[0144] The feature extraction module is configured to obtain a customs clearance image-text pair, and extract original features of the customs clearance image including multi-scale features;

[0145] The adaptive fusion module is configured to take the first N layers of features in the multi-scale features as shallow features and the remaining layers of features as deep features, respectively, to generate shallow fusion features and deep fusion features after weighted fusion of the shallow features and the deep features, to obtain a pooling feature after multi-scale pooling operation on the deep fusion features, to splice the shallow fusion features, the deep fusion features and the pooling feature to generate enhanced features, and to integrate the enhanced features and the original features to obtain visual features.

[0146] The training module is configured to align the visual features with the text features in modalities, perform two-stage training on the detection model based on the obtained aligned multi-modal features, and obtain a contraband detection result for a to-be-detected customs clearance image by using the trained detection model.

[0147] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules can be executed in a computer system such as a set of computer executable instructions as part of a system.

[0148] In more embodiments, there are also provided:

[0149] An electronic device includes a memory and a processor, and computer instructions stored on the memory and running on the processor, when the computer instructions are executed by the processor, the method described in Embodiment 1 is completed. For brevity, it will not be described here.

[0150] It should be understood that in the embodiments, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0151] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0152] A computer readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the method described in Embodiment 1 is completed.

[0153] The method in Embodiment 1 can be directly embodied as hardware processor execution completion, or executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0154] A computer program product includes a computer program, which is executed by the processor to implement the method described in Embodiment 1.

[0155] The present application also provides at least one computer program product tangibly embodied on a non-transitory computer readable storage medium. The computer program product includes computer executable instructions, for example, instructions embodied in program modules, executed by devices at the target real or virtual processor to perform the processes / methods as described above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In various embodiments, the functionality of program modules can be combined or split between program modules as desired. Machine executable instructions for program modules can be executed within a local or distributed device. In a distributed device, program modules can be located in local and remote memory storage devices.

[0156] Computer program code for carrying out operations of the present application can be written in one or more programming languages. The computer program code can be provided to a processor of a general or special purpose computer, or other programmable data processing apparatus, to produce a machine, such that the code, which can be executed via the processor of the computer or other programmable data processing apparatus, produces a result. The computer program code can be embodied on a computer readable medium, which can be a memory or a storage device. The computer program code can also be embodied in the form of a software package comprising a plurality of devices, which can be executed on and / or transferred via a computer or other programmable data processing apparatus.

[0157] In the context of the present application, the computer program code or related data can be carried by any suitable carrier, to enable the device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, and the like. Examples of signals can include electrical, optical, radio, sound or other forms of propagated signals, such as carrier waves, infrared signals, and the like.

[0158] Those skilled in the art can understand that the units and algorithm steps of the examples described in conjunction with the embodiments can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0159] The above describes the specific embodiments of the present application in conjunction with the accompanying drawings, but is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without inventive labor are still within the scope of protection of the present application.

Claims

1. A customs contraband detection method based on multimodal adaptive fusion, characterized in that, include: Obtain customs clearance image-text pairs and extract raw features from the customs clearance image, including multi-scale features; The first N layers of features in the multi-scale feature set are designated as shallow features, and the remaining layers are designated as deep features. The shallow and deep features are weighted and fused to generate shallow fused features and deep fused features, respectively. The deep fused features are then subjected to multi-scale pooling to obtain pooled features. The shallow fused features, deep fused features, and pooled features are concatenated to generate enhanced features. The enhanced features and the original features are then integrated to obtain the visual features. Specifically, shallow and deep features are extracted from the outputs of all hidden layers of the visual encoder and input into the adaptive dynamic channel fusion (ADCI) module. The generated enhanced feature sequence is integrated with the original image feature sequence along the channel dimension, achieving multi-scale, multi-level feature fusion and enhancement, generating richer, more structured high-dimensional visual feature sequences. The Adaptive Dynamic Channel Fusion (ADCI) module integrates the generated enhanced feature sequence with the original image feature sequence along the channel dimension. The specific steps are as follows: Extract shallow and deep features from the hidden layer of the visual encoder; In ADCI, adaptive weights are calculated based on the L2 norm of the features and then weighted and fused to generate shallow and deep fused features. The weights for each layer are as follows: in, It is the first Weights of shallow features; It is the first Weights of deep features; It is the first The shallow feature in the first The position and the first Values ​​in each dimension; It is the first The deep feature in the first The position and the first Values ​​in each dimension; For sequence length, To hide dimensions; Then, shallow adaptive weights are generated using Softmax normalization. and deep adaptive weights : Finally, shallow fusion features are generated using weighted fusion. and deep fusion features ; In ADCI, multi-scale pooling operations are applied to the deep fusion features to generate deep pooled features. In ADCI, shallow fusion features, deep fusion features, and deep pooling features are concatenated along the channel dimension to generate an enhanced feature sequence. The concatenation formula is as follows: in, It is a shallow fusion feature; It is a deep fusion feature; This is a feature of deep pooling; This refers to splicing along the last dimension; In ADCI, the enhanced feature sequence and the original image feature sequence are integrated along the channel dimension to output the final high-dimensional visual feature sequence. The integration formula is as follows: in, It is the original image feature sequence; It is a feature sequence enhanced by the ADCI module; This refers to splicing along the last dimension; For high-dimensional visual feature sequences, a two-sided soft matching strategy is introduced to reduce the length of the feature sequence; Visual features and text features are modally aligned. Based on the obtained aligned multimodal features, the detection model is trained in two stages. The trained detection model is then used to test the customs clearance image to obtain the detection result of contraband.

2. The customs contraband detection method based on multimodal adaptive fusion as described in claim 1, characterized in that, After obtaining the customs clearance image, divide the customs clearance image into a×b cropping blocks, then the total number of visual tokens is L=(a×b+1)×T; when the total number of visual tokens L exceeds the set token number threshold τ, adjust the token number and select the optimal space configuration. ； in, T is the number of tokens in each cropped block after adjustment, and T is the original number of tokens in each cropped block; a and b represent the number of columns and rows of the division; a×b+1 is the total number of cropped units calculated by adding the original image to the total number of cropped blocks.

3. The customs contraband detection method based on multimodal adaptive fusion as described in claim 1, characterized in that, The process of performing multi-scale pooling operations on deep fusion features includes: Deep fusion features Perform one-dimensional average pooling, with the permutation dimension being... Transform the shape ; Application pooling operations The output shape is ,in ; Through linear interpolation operations After adjusting the sequence length of the deep pooling features to L, adjust the dimensional order to... ; For batch size, and For sequence length, To hide the dimension.

4. The customs contraband detection method based on multimodal adaptive fusion as described in claim 1, characterized in that, For visual feature representation, a two-sided soft matching strategy is introduced, using cosine similarity to measure token similarity, matching and fusing the token pairs with the highest similarity to generate a new token; Specifically, this includes: dividing all tokens equally into two sets A and B; for each token in set A, using cosine similarity, finding the token in set B that is most similar to it, and connecting them with edges; after completing the matching, merging the original tokens by weighted average to generate a new token; By combining paired image-text pairs with corresponding instruction templates, a visual instruction tuning dataset for model fine-tuning is constructed. Furthermore, a two-layer mapping layer maps visual features to a semantic space consistent with text features to achieve modal alignment. The two-stage training process includes: the first stage is the cross-modal mapping optimization stage, where the parameters of the visual encoder and language encoder are frozen, and the two-layer mapping layer is trained to optimize the mapping relationship between visual features and text features, achieving preliminary alignment between modalities; the second stage is the visual instruction tuning stage, where the parameters of the visual encoder and language encoder are unfrozen, and the overall model is trained together with the two-layer mapping layer.

5. The customs contraband detection method based on multimodal adaptive fusion as described in claim 1, characterized in that, For length of The sequence, the target answer probability Defined as: ； in, These are the visual features after modal alignment; These are textual features of the instruction template; It contains all the tokens in the instruction template. It is the first part of the answer. Each of the following tokens represents the token used to predict the current answer. At that time, the language encoder has already processed the instruction and answer context; This is represented as the language encoder predicting each token of the answer one by one using an autoregressive method. Each prediction references visual features. Command Token and the generated answer token Calculate the probability of the current token. Finally, the probabilities of all tokens are multiplied together to obtain the probability of the entire answer; the final result includes the name of the contraband and the location of the bounding box.

6. A customs contraband detection system based on multimodal adaptive fusion, employing the detection method as described in any one of claims 1-5, characterized in that, include: The feature extraction module is configured to acquire customs clearance image-text pairs and extract raw features of the customs clearance image, including multi-scale features. The adaptive fusion module is configured to use the first N layers of features in the multi-scale feature set as shallow features and the remaining layers as deep features. The shallow and deep features are weighted and fused to generate shallow fused features and deep fused features respectively. The deep fused features are then subjected to multi-scale pooling to obtain pooled features. The shallow fused features, deep fused features and pooled features are concatenated to generate enhanced features. The enhanced features and the original features are then integrated to obtain visual features. The training module is configured to perform modal alignment between visual features and text features. Based on the obtained aligned multimodal features, the detection model is trained in two stages. The trained detection model is then used to test the customs clearance image to obtain the contraband detection result.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-5.

Images