Tobacco manufacturing foreign matter detection method based on multi-modal lightweight dynamic network

Abstract

The application discloses a kind of based on multimodal light weight dynamic network's tobacco manufacturing foreign matter detection method, mainly for the detection problem of metal scrap, plastic film, fiber and other foreign matters mixed in tobacco, propose the multimodal dynamic network of fusion tobacco RGB image and near-infrared spectrum image, including: constructing dynamic channel reorganization encoder, using asymmetric channel split to RGB branch, multi-band weighted processing is carried out to near-infrared branch, and temperature perception sparse strategy is introduced to dynamically adjust calculation amount;Construct multi-modal feature dynamic routing, by learnable routing matrix fusion tobacco image, near-infrared and vibration spectrum features;Lightweight spatiotemporal detection head is constructed, combined with depth separable convolution and LSTM is carried out motion compensation time series modeling, and structural pruning and dynamic quantization are used to carry out hardware adaptive model compression.The application greatly compresses algorithm volume while maintaining high detection accuracy, realizes the efficient real-time detection of multiple types of foreign matters under the complex working condition of tobacco manufacturing line.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, and in particular to a method for detecting foreign matter in tobacco processing based on a multimodal lightweight dynamic network, which is especially suitable for the online automatic detection and removal of foreign matter mixed in tobacco shreds on a tobacco processing production line. Background Technology

[0002] As the core raw material of the tobacco industry, the quality of tobacco leaves directly affects the quality of cigarette products and the industry's economic benefits. During the tobacco processing, foreign objects such as metal shavings, plastic film, fibers, cardboard, and hemp rope are often mixed in during the tobacco processing line. These impurities not only reduce the quality of the finished cigarettes but may also damage high-precision production equipment such as shredders, causing serious economic losses.

[0003] Traditional manual sorting methods are inefficient and completely unsuitable for the processing capacity of several tons per hour on modern production lines. Currently, the tobacco industry mainly uses the following foreign object removal technologies: First, laser sorting technology, which identifies foreign objects by utilizing the difference in reflectivity between foreign objects and tobacco leaves under laser irradiation. However, this technology has an identification rate of less than 50% for transparent and light-colored foreign objects, making it difficult to meet increasingly stringent quality control requirements. Second, traditional machine vision methods have an extremely high error rate in the high-frequency vibration environment of the tobacco processing line's vibrating trough, resulting in the incorrect rejection of a large number of qualified tobacco leaves, which seriously affects production efficiency.

[0004] Therefore, the industry has proposed a visual detection and removal scheme for foreign objects in tobacco residue. However, it only uses a single RGB image mode, which results in limited ability to identify foreign objects that are similar in color to tobacco leaves. Another method for removing foreign objects from tobacco residue uses a combination of laser scanning, electromagnetic field detection and transmission scanning to detect foreign objects. However, the system is complex and the equipment cost is high. It does not address the improvement of detection robustness under material conveying vibration conditions.

[0005] With the rapid development of machine vision technology, intelligent sorting and detection based on image recognition has become a research hotspot. Although vision technology has advantages such as non-contact, high efficiency, and digitalization compared to traditional methods, it still faces several key technical challenges in practical applications:

[0006] (1) If the RGB image and the near-infrared image are simply stitched together in terms of channel dimension, it will lead to feature conflict and make the model convergence difficult. Especially when the color of the foreign object is similar to that of the tobacco leaf (such as cardboard, hemp rope, black plastic, etc.), the recognition rate based on the RGB image drops sharply to 10%-15%. At present, the industry has not fully explored the complementary information between different modalities.

[0007] (2) The silk production line has complex working conditions such as high vibration frequency, conveyor belt speed fluctuation and ambient light change. The fixed structure of the neural network cannot cope with these dynamic changes. In particular, vibration factors will cause image blurring, which will significantly increase the false detection rate of the existing algorithm.

[0008] (3) The model parameters of mainstream detection networks, such as Faster R-CNN, reach 41.2MB, which far exceeds the storage and computing capacity of industrial embedded devices, making it difficult to promote and apply in tobacco companies. At the same time, existing lightweight methods often sacrifice detection accuracy when compressing models, and cannot simultaneously take into account real-time performance and accuracy.

[0009] (4) Existing detection methods only focus on foreign object identification at the image level. The detection results fail to form a closed-loop control with the key operating parameters of the embedded processing device, such as power and temperature. The detection performance degrades significantly when the device is overheated or under high load, and it lacks the ability to adaptively adjust to the device status.

[0010] As can be seen from the above analysis, there is an urgent need in the tobacco processing industry to develop a foreign object detection method that can integrate multimodal information, dynamically adapt to complex working conditions, be lightweight, and have high detection accuracy, so as to meet the tobacco industry's comprehensive needs for real-time, accurate, and low-cost deployment of foreign object detection. Summary of the Invention

[0011] In view of the above, the present invention aims to provide a method for detecting foreign objects in tobacco processing based on a multimodal lightweight dynamic network, which is used to perform high-precision, real-time online detection of various foreign objects such as metal shavings, plastic film, fibers, cardboard, and hemp rope in tobacco processing production lines.

[0012] The technical solution adopted in this invention is as follows:

[0013] A method for detecting foreign matter in tobacco processing based on a multimodal lightweight dynamic network is used to detect foreign matter mixed in tobacco shreds on a tobacco processing production line, comprising the following steps:

[0014] RGB and near-infrared spectral images of tobacco shreds on the tobacco processing conveyor belt are acquired simultaneously using industrial cameras.

[0015] Specifically, an industrial camera and a near-infrared spectral camera are positioned above the vibrating conveyor belt before the tobacco cutting process in the tobacco processing line. These cameras simultaneously capture images of the continuously flowing tobacco on the conveyor belt, obtaining RGB visible light and near-infrared (NIR) spectral images of the tobacco to be inspected. Preferably, the industrial camera is a high-resolution color camera, and the near-infrared spectral camera covers three characteristic bands: 900nm, 1200nm, and 1500nm. In the scenario of foreign object detection in tobacco, RGB images can effectively capture high-frequency details such as color, texture, and edge contours between the foreign object and the tobacco, and have good distinguishing ability for foreign objects with obvious color differences from tobacco leaves, such as wood chips and chicken feathers. However, for foreign objects with colors similar to tobacco leaves, such as cardboard, hemp rope, and black plastic, effective identification is difficult to achieve using only RGB images. Near-infrared spectral images can reflect the differences in spectral absorption characteristics between tobacco leaves and foreign objects at the molecular vibrational energy level: cellulose, chlorophyll, and water in tobacco leaves have characteristic absorption peaks in the near-infrared band, while the spectral characteristics of foreign objects such as metal shavings, plastic films, and fibers differ significantly from those of tobacco leaves, especially in the 900nm-1500nm band. Therefore, by simultaneously acquiring image data from both modalities, a comprehensive characterization of foreign objects can be achieved from two complementary dimensions: color texture and spectral characteristics, effectively solving the problem of identifying foreign objects with similar colors.

[0016] The RGB image and near-infrared spectral image are respectively input into the RGB branch and near-infrared branch of the dynamic channel reconstruction encoder. The RGB branch uses an asymmetric channel splitting strategy to perform channel compression and reconstruction on the RGB image feature map. The near-infrared branch uses a multi-band splitting strategy to perform multi-band weighted processing on the near-infrared spectral image feature map and introduces a hardware-aware sparsity strategy for dynamic sparsification processing, and outputs RGB encoded features and near-infrared encoded features respectively.

[0017] The core of the above method lies in designing a differentiated encoding strategy based on the essential differences in the information characteristics contained in the RGB and near-infrared modes, rather than using a unified encoder. At the same time, it innovatively incorporates the temperature status of the embedded processing device into the dynamic control of the encoding process.

[0018] Specifically, for the RGB branch (high-frequency detail extraction), an asymmetric channel splitting strategy is adopted. This strategy is based on the inventors' insight that in the tobacco foreign object detection scenario, not all channels in the RGB image contribute equally to foreign object detection—some channels mainly encode the color and texture information of the tobacco leaf itself, while other channels are more sensitive to foreign object edges and texture differences. By dynamically filtering and compressing the channels, channels that contribute significantly to foreign object detection can be retained for full computation, while channels that contribute less are compressed into statistical information, thereby reducing computational load while preserving key detection information. The specific operation is as follows: the input feature map is split according to a dynamic compression ratio: the first k channels are retained as active channels for subsequent computation; the remaining channels are compressed into mean channels using global average pooling; and finally, the output is reassembled by direct concatenation. Next, the reconstructed feature map is further processed using lightweight residual blocks based on depthwise separable convolution for feature extraction, and the feature representation is enhanced by channel attention mechanism. Among them, depthwise separable convolution decomposes the standard convolution into two steps: channel-wise convolution and pointwise convolution, which can reduce the computational cost to about 1 / 9 of the standard convolution, significantly reducing the computational overhead of the encoder.

[0019] For the near-infrared branch (spectral feature extraction), since the spectral absorption characteristics of tobacco leaves and foreign objects differ significantly in different near-infrared bands, different bands need to be processed separately to fully extract the identification information in the spectral dimension. Specifically, the near-infrared spectral image is split into several bands; each band is independently feature-extracted through convolution, and then weighted and summed using learnable weights. The learnable weights allow the network to automatically learn the relative importance of each band for foreign object detection based on training data. For example, the absorption characteristics of common plastic films in tobacco foreign objects may be most significant in the 1200nm band, while the reflection characteristics of metal scraps are more prominent in the 900nm band. After weighting the features across multiple bands, a spectral attention mechanism is used to enhance the feature representation in the spectral dimension, and a spatial enhancement method is used to improve the spatial resolution of the foreign object location. Finally, the features are integrated through direct stitching.

[0020] In the aforementioned encoding process, a temperature-sensing sparsity strategy is further introduced. The technical concept behind this strategy is that during long-term continuous operation of the embedded processing equipment in the fabrication workshop, the chip temperature gradually increases. When the temperature exceeds a safe threshold, it may lead to frequency reduction or increased detection latency. By incorporating equipment temperature information into the encoding process, an adaptive match between computational load and equipment thermal state is achieved. Specifically, the sparsity rate is dynamically adjusted based on the real-time temperature of the embedded processing equipment, and the encoded features are sparsified using a Bernoulli sampling matrix. The generation probability of the Bernoulli sampling matrix is ​​controlled by the equipment temperature: when the equipment temperature is low (e.g., far below the threshold), only about 30% of the features are sparsified, with most channels participating in the calculation to ensure optimal detection accuracy; while when the equipment temperature gradually rises to near or above the threshold, about 70% of the features are sparsified, thereby significantly reducing computational load and chip heat generation. The upper limit of the sparsity probability can be set as needed to constrain this, ensuring that even under extreme high-temperature conditions, a certain number of feature channels remain active, avoiding a precipitous drop in detection accuracy. The aforementioned temperature-adaptive dynamic masking mechanism enables the model to automatically achieve the optimal balance between detection accuracy and computational efficiency based on the actual operating status of the equipment.

[0021] Continuing from the previous section, the RGB encoding features, near-infrared encoding features, and the vibration spectrum features of tobacco transmission collected by the vibration sensor are input into the multimodal feature dynamic routing module. The three modal features are weighted and fused through a learnable routing matrix to output a fused feature map.

[0022] This step is one of the core concepts of this invention, specifically addressing the problems of feature conflicts and low fusion efficiency caused by simple splicing of multimodal features in existing technologies. It uses the operating status of the tobacco conveying equipment as a fusion weight adjustment factor. It should be noted that the vibration of the vibrating groove in the tobacco processing line not only causes image blurring, but its vibration spectrum itself also contains valuable information: the vibration response characteristics of foreign objects (such as metal blocks or plastic sheets) on the groove differ from those of tobacco. The spectral data collected by the vibration sensor can provide additional clues for foreign object identification from the perspective of physical motion. Therefore, this invention incorporates the vibration spectrum as a third modality into the fusion framework. The specific processing includes the following sub-steps: routing matrix initialization, dynamic updating of the routing matrix, and cross-modal feature fusion.

[0023] Then, the aforementioned fused feature map is input into the lightweight spatiotemporal detection head, which sequentially performs spatial feature extraction through depthwise separable convolution, performs temporal modeling through an LSTM network, and performs temporal alignment through a motion compensation module when historical frames exist. Finally, the detection result includes the foreign object's position coordinate offset, confidence level, and class probability.

[0024] As previously analyzed, image blurring and foreign object displacement caused by silk-thread vibration are key factors limiting detection accuracy. Traditional single-frame detection methods ignore the temporal correlation between consecutive frames, making them prone to false detections and missed detections in vibration environments. This invention improves detection robustness in vibration environments by introducing temporal modeling and motion compensation mechanisms, fully utilizing inter-frame information, while employing a lightweight design to control the parameters of the detection head. The specific processing includes the following sub-steps: spatial feature extraction, temporal modeling, motion compensation, and detection output.

[0025] Finally, the location coordinates of the foreign object detected are mapped onto the physical coordinate system of the conveyor belt. The time it takes for the foreign object to reach the bottom of the removal device is calculated in combination with the speed of the conveyor belt. When the foreign object passes by, the high-pressure nozzle and adsorption component are triggered to accurately capture and remove it from the tobacco.

[0026] In at least one possible implementation of the above scheme, the present invention also includes, during the inference deployment phase, hardware-adaptive model compression of the trained model to enable the model to run efficiently on embedded chips with different computing power. Specifically, this involves structured pruning, which prunes the entire channel at a time. The compressed model can still fully utilize the hardware parallel computing capabilities on a general inference framework. Additionally, dynamic range quantization can significantly reduce model storage overhead and memory bandwidth requirements.

[0027] The pruning ratio involved in the above pruning process can be determined according to the type of embedded chip architecture used in the actual application. For example, for computing devices with high computing power, a lower pruning ratio can be used to maintain high accuracy; while for computing devices with relatively limited computing power, a higher pruning ratio can be used to obtain lower computational load and faster inference speed.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] First, addressing the challenge of low recognition rates for foreign objects with similar colors in tobacco foreign object detection, this invention designs a multimodal detection framework that integrates RGB images and near-infrared spectral images. RGB images provide high-frequency details such as the color, texture, and edge contours of the foreign object, while near-infrared spectral images provide the differences in spectral absorption characteristics between tobacco leaves and foreign objects at the molecular vibrational energy level. This complementary fusion of the two modes effectively improves the recognition rate of difficult-to-detect foreign objects such as cardboard, hemp rope, and black plastic, for which traditional RGB methods only achieve a recognition rate of 10%-15%.

[0030] Secondly, to address the issue of image blurring and decreased detection accuracy caused by vibration of the spinning line, this invention introduces an LSTM temporal modeling and motion compensation module into the detection head. By capturing the motion trajectory information of foreign objects between consecutive frames and compensating for spatial offsets caused by vibration, the robustness of detecting motion-blurred foreign objects under vibration conditions is effectively improved, and the missed detections and false detections caused by image blurring are reduced.

[0031] Third, this invention designs a dynamic control mechanism for computing hardware awareness, including a temperature-adaptive sparsity strategy in the dynamic channel reassembly encoder and a device state adjustment factor in the dynamic routing of multimodal features. These mechanisms enable the model to automatically adjust the computational load and modal fusion ratio according to the real-time temperature, memory, and power consumption status of the embedded device, achieving an optimal balance between detection accuracy and computational efficiency, and ensuring stability during long-term continuous operation.

[0032] Fourth, this invention combines multiple lightweight technologies such as depthwise separable convolution, asymmetric channel splitting, structured pruning, and dynamic quantization to significantly compress the number of model parameters, enabling the model to perform real-time inference on embedded devices commonly used by tobacco companies. This reduces system hardware costs and deployment barriers, and facilitates the promotion and application of the proposed solution.

[0033] Fifth, this invention realizes an end-to-end closed-loop detection process from image acquisition, multimodal coding, feature fusion, spatiotemporal detection to foreign object removal, and incorporates the vibration spectrum characteristics of tobacco conveying into the multimodal fusion framework, providing additional clues for foreign object identification from the dimension of physical motion, thus forming a complete and efficient intelligent detection solution for foreign objects in tobacco processing. Attached Figure Description

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:

[0035] Figure 1 The overall flowchart of the foreign object detection method for yarn fabrication based on a multimodal lightweight dynamic network provided in the embodiments of the present invention;

[0036] Figure 2 This is a schematic diagram of the dynamic channel reassembly encoder structure provided in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the multimodal feature-based dynamic routing module structure provided in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the lightweight spatiotemporal detection head structure provided in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the hardware adaptive model compression process provided in an embodiment of the present invention;

[0040] Figure 6 Comparison of the original acquired images provided in the embodiments of the present invention with the foreign object detection implemented by the solution of the present invention. Detailed Implementation

[0041] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0042] This invention proposes an embodiment of a foreign matter detection method for filament fabrication based on a multimodal lightweight dynamic network. Specifically, as follows: Figure 1 As shown, the overall flow of the foreign object detection method for filament fabrication based on multimodal lightweight dynamic networks of the present invention is as follows:

[0043] First, an industrial camera and a near-infrared spectroscopy camera are installed above the vibrating conveyor belt before the tobacco cutting process in the tobacco processing line to simultaneously acquire RGB and near-infrared spectral images of the tobacco shreds. At the same time, a vibration sensor is installed below the vibrating trough to acquire vibration spectrum signals in real time.

[0044] Secondly, the acquired RGB and near-infrared spectral images are respectively sent to the dynamic channel reconstruction encoder (reference). Figure 2 (As shown) Differential encoding processing is performed. The dynamic channel reconstruction encoder contains two parallel processing paths: an RGB branch and a near-infrared branch. Each branch includes a channel splitting unit, a depthwise separable convolutional residual block, and a hardware-aware sparse unit. After encoding processing, RGB encoded features and near-infrared encoded features are output respectively.

[0045] Then, the RGB encoded features, near-infrared encoded features, and vibration spectrum features are input together into the multimodal feature dynamic routing module (see reference). Figure 3 (As shown). This module first projects and maps the features of each modality using 1×1 convolutions, while simultaneously acquiring three state parameters of the embedded processing device: temperature T, memory M, and power consumption P. A routing weight vector α is generated through a fully connected layer to dynamically update the initial routing matrix. The updated routing matrix is ​​then used to weight and fuse the projected features of each modality, outputting a fused feature map.

[0046] Next, the fused feature map is input into the lightweight spatiotemporal detection head (see reference). Figure 4(As shown). The detection head first extracts spatial features through depthwise separable convolutional layers, then reshapes the extracted spatial features into a sequence form and inputs it into an LSTM network for temporal modeling. When historical frame data is available, a motion compensation module performs temporal alignment of the historical frame features. After format restoration, the temporal features output by the LSTM are used to predict a 6-dimensional detection output through a 1×1 convolutional layer: 4 bounding box coordinate offsets, 1 target confidence score, and 1 class probability. The precise location of the foreign object in the image is obtained by decoding the coordinate offsets.

[0047] Finally, the pixel coordinates of the foreign objects in the detection results are mapped to the physical coordinates of the conveyor belt, and the pneumatic removal device is controlled to remove the detected foreign objects from the tobacco.

[0048] Before deploying the model, hardware-adaptive model compression is also required (see reference). Figure 5 As shown in the figure, select the appropriate pruning ratio and quantization parameters according to the target chip architecture to perform structured pruning and dynamic range quantization in sequence, so that the compressed model size is suitable for the storage and computing capabilities of the embedded device.

[0049] Reference Figure 2 The implementation of dynamic channel re-encoding specifically includes: the dynamic channel re-encoding encoder performs specific processing on the RGB and near-infrared modes.

[0050] In the RGB branch, input feature map First, we enter the asymmetric channel splitting unit. Setting the dynamic compression ratio α = 0.6, the first k = |0.6C| channels are retained as active channels. It directly participates in the subsequent calculation of depthwise separable convolutional residual blocks; the remaining channels Global average pooling is used to compress the data into a single-channel mean representation. The active channel retains high-frequency details such as texture, edge, and color contrast that contribute most to foreign object detection in the original feature map, while the mean channel retains the overall statistical information of the remaining channels with extremely low information storage cost, avoiding information loss caused by directly discarding channels. The active and mean channels are concatenated along the channel dimension and then fed into a lightweight residual block. The lightweight residual block consists of depthwise separable convolutional layers, batch normalization layers, ReLU activation functions, and channel attention modules stacked sequentially. Depthwise separable convolution reduces the number of parameters and computational cost of standard convolution to about 1 / 9; the channel attention module learns the importance weights of each channel through a classic SE structure of global average pooling, fully connected layer compression-expansion, and sigmoid gating, adaptively enhancing the feature channels related to foreign object detection.

[0051] In the near-infrared branch, the input feature map is split into three bands: 900nm, 1200nm, and 1500nm, resulting in three band sub-feature maps. Each band sub-feature map is then independently extracted using a 5×5 convolution kernel. It's worth noting that the 5×5 kernel has a larger receptive field than the common 3×3 kernel, making it suitable for capturing broader spatial spectral features in near-infrared images. Finally, a weighted sum is performed using three learnable weights. In practice, the three learnable weights can be initialized to 0.33 and automatically optimized through backpropagation during training. Taking the common plastic film foreign matter in tobacco as an example, the plastic film usually exhibits a strong characteristic absorption peak in the 1200nm band, while its absorption characteristics are weaker in the 900nm band. The trained model will automatically assign a higher weight to the 1200nm band. For the weighted features, a spectral attention mechanism and a spatial enhancement method are applied sequentially. The spectral attention mechanism calculates the attention weights of each band along the spectral dimension, strengthening the features of important bands; the spatial enhancement method enhances the spatial response of the foreign matter location through self-attention operations in the spatial dimension. Finally, the enhanced multi-band features are spliced ​​and integrated along the channel dimension to obtain the near-infrared coded features.

[0052] The hardware-aware sparsity strategy is executed at the feature output terminals of both modalities. Taking the temperature change of an embedded processing device in a silk-making workshop as an example: when the embedded device is first started, the temperature is 30℃, at which point σ(30-threshold 60)=σ(-30)≈0, the sparsity probability is about 0.3, and only about 30% of the feature channels are randomly sparsified, and the model runs in full precision state; after the device runs continuously for two hours, the temperature rises to 60℃, at which point σ(60-60)=σ(0)=0.5, the sparsity probability is 0.3+0.4×0.5=0.5, about 50% of the feature channels are sparsified, and the computational load is reduced by about 50%; when the temperature continues to rise to 70℃, σ(70-60)=σ(10)≈1, the sparsity probability is 0.3+0.4×1=0.7, reaching the upper limit of 0.7, the computational load is greatly reduced, and the device power consumption and heat generation decrease accordingly, forming a closed-loop adaptive adjustment of temperature and computational load. This mechanism effectively avoids performance degradation or system instability of embedded devices due to workshop environment and overheating of internal components during long-term operation, ensuring continuous and reliable operation of foreign object detection on the yarn production line.

[0053] Reference Figure 3 The implementation of multimodal feature dynamic routing specifically includes the multimodal feature dynamic routing module processing three modal features: RGB encoding features, near-infrared encoding features, and vibration spectrum features.

[0054] The initial routing matrix for dynamic routing is an identity matrix. In the initial training phase, this means that each modal feature is only multiplied by its own weight before being fused, and there is no cross-modal information interaction. This design helps the model establish stable single-modal feature representations in the early stages of training, avoiding training instability caused by noise interference introduced by randomly initialized cross-modal routes.

[0055] As training progresses and forward inference continues, the device state vector s=[T norm M norm ,P norm (where T) norm M norm ,P norm (Normalized values ​​for temperature, memory, and power consumption, respectively) are input to the fully connected layer, and a three-dimensional weight vector α is generated through the nonlinear mapping of the fully connected layer. The weights of the fully connected layers are obtained through joint optimization learning during training, enabling the model to automatically discover the mapping relationship between device states and the optimal modality fusion ratio. Then, the routing matrix is ​​updated through element-wise multiplication. The updated routing matrix no longer has 1s on the diagonal and 0s on the off-diagonal, indicating that controllable information exchange has begun between the modalities.

[0056] During the fusion phase, the features of each modality are first projected onto a unified dimension through a 1×1 convolution, and then weighted and summed according to the weights in the routing matrix: Since the routing weight α is dynamically generated by the device state, when the device memory usage is high, It may automatically decrease. The memory pressure is reduced by increasing the dimensionality of vibration spectrum features (because the dimensionality of vibration spectrum features is much lower than that of image features, resulting in lower computational overhead). This mechanism, which uses hardware status as a routing adjustment factor, enables the model to not only adaptively fuse based on the content of the input data, but also to adaptively adjust based on the state of the operating environment, achieving intelligent fusion of data and hardware dual perception.

[0057] Reference Figure 4 The implementation of lightweight spatiotemporal detection is specifically as follows:

[0058] In the spatial feature extraction stage, feature maps are fused. The system first enters the depthwise separable convolutional layer. Depthwise convolution consists of concatenated depthwise and pointwise convolutions: depthwise convolution assigns an independent 3×3 kernel to each input channel, extracting only the spatial information of that channel without inter-channel interaction; pointwise convolution uses a 1×1 kernel to linearly combine the output channels of all depthwise convolutions, achieving inter-channel information fusion. For a feature map with C input channels, C' output channels, and K kernel size, the computational cost of standard convolution is H×W×C×C'×K×K, while the computational cost of depthwise separable convolution is H×W×C×K×K+H×W×C×C', with a ratio of approximately 1 / C'+1 / K². When C'=64 and K=3, the ratio is approximately 1 / 8.9. This significant reduction in computational cost enables the detection head to operate in real-time on embedded devices.

[0059] Spatial features After being reshaped into a sequence [B, HW, C], the data is input into an LSTM network. The LSTM's gating mechanism, through the synergistic action of forget gates, input gates, and output gates, selectively remembers and forgets inter-frame information. In the scenario of detecting foreign objects in tobacco, the spatial positional changes of foreign objects between three consecutive frames can provide crucial temporal clues for detection: normal tobacco, being relatively light, exhibits a regular pattern of jumping up and down but with small horizontal displacement on the vibration groove; denser foreign objects such as metal scraps exhibit relatively independent trajectories during vibration, possibly remaining stationary or sliding horizontally. The LSTM learns these differences in motion patterns through its gating mechanism, encoding temporal information into the hidden state. middle.

[0060] The motion compensation module is activated when historical frame data is detected. Offset estimation uses the inter-frame difference method. This calculation, which takes the average value along the channel dimension and then differs it with the current frame features, has low computational complexity and is suitable for embedded real-time processing scenarios. Taking the typical vibration period of tobacco shreds on the vibrating groove of the tobacco processing line (approximately 50ms) as an example, the maximum displacement of the tobacco shreds on the image between two consecutive frames (assuming a frame rate of 30fps and a frame interval of approximately 33ms) is about 3-5 pixels. The motion compensation module aligns the features of historical frames to the current frame through bilinear sampling, controlling the displacement error to the sub-pixel level (less than 0.5 pixels), effectively eliminating the feature space misalignment caused by vibration.

[0061] Motion-compensated aligned LSTM output timing features It is restored to a spatial format of [H, W, C / 2], and then mapped to a 6-channel output through a 1×1 convolutional layer: the first 4 channels represent the center coordinate offset of the bounding box (t). x ,t y ) and width and height offset (t) w ,th The 5th channel is activated by Sigmoid to output the confidence score, and the 6th channel is activated by Softmax to output the class probability.

[0062] Reference Figure 5 The specific implementation of hardware adaptive model compression also includes performing hardware adaptive compression after the model training is completed and before it is deployed to an embedded device.

[0063] (1) In the structured pruning stage, the importance score of each input channel of the convolutional layer to be pruned is first calculated. The importance score is obtained by averaging the absolute values ​​of the activation values ​​at all spatial locations of the channel: This metric is based on the assumption that "channels with larger mean absolute activation values ​​contribute more to the output of the next layer," and it is simple to calculate and naturally compatible with the forward propagation process of the model.

[0064] Then, the quantile threshold is determined according to the preset pruning ratio `prune_ratio`. For example, when `prune_ratio=0.3`, the 30th percentile after sorting the importance scores is taken as the threshold. Channels with importance scores below this threshold are judged as unimportant channels and are set to zero using a binary mask. Channels set to zero do not participate in subsequent inference calculations, which is equivalent to being pruned. A key advantage of structured pruning is that the remaining channels after pruning are still regular tensor structures, which can be efficiently computed in parallel on the CUDA cores of the GPU or the matrix multiplication unit of the NPU, without incurring the additional overhead of sparse matrix operations.

[0065] (2) In the dynamic quantization stage, low-bit-width quantization is performed on the model weights and activation values ​​after pruning. Taking INT8 quantization as an example, the maximum value is calculated for the weight tensor of each layer. and minimum value scaling factor scale=( ~- ) / (2 8 -1)=( ~- The value is calculated as 1 / 255, and then each floating-point value is mapped to a combination of an 8-bit integer and a scaling factor. During inference, matrix multiplication and accumulation operations are performed using INT8 integers, reducing model storage space and memory bandwidth requirements by approximately 75% compared to FP32. On chips with built-in NPUs, such as the RK3588, INT8 operations receive hardware-level acceleration, increasing inference speed by 2 to 4 times.

[0066] For different chip architectures, the configuration strategies for pruning ratio and quantization bit depth can be referenced as follows: For computing devices with a computing power of 275 TOPS, setting prune_ratio=0.2-0.3 and b=8 can achieve more than 3 times inference acceleration while keeping the mAP accuracy loss below 1%; For devices with a computing power of 6 TOPS, prune_ratio=0.4-0.5 and b=8 can be set, compressing the model size to less than 20% of the original model, and the inference speed can meet the real-time detection requirements of the filament fabrication line at 30fps.

[0067] Finally, refer to Figure 6 The diagram illustrates a comparison of different foreign object detection methods. The left column (a) shows the original images captured during the tobacco conveying process, displaying the raw RGB images of tobacco and foreign objects mixing and flowing on the vibrating conveyor belt of the tobacco processing line. The foreign objects are difficult to distinguish with the naked eye compared to the tobacco. The right column (b) shows the foreign object detection effect after processing using the method of this invention, with the detection frame precisely marking the location of the foreign objects. Verification results show that the above-mentioned method of this invention can effectively detect foreign objects with a color similar to tobacco leaves, meeting the detection requirements of actual production in the tobacco processing workshop.

[0068] In this invention, when directional terms are mentioned, they are relative concepts based on the embodiments. Furthermore, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0069] The above description of the structure, features, and effects of the present invention is based on the embodiments shown in the figures. However, the above are only preferred embodiments of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched by those skilled in the art to form a variety of equivalent solutions without departing from or changing the design concept and technical effects of the present invention. Therefore, the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A method for detecting foreign matter in tobacco processing based on a multimodal lightweight dynamic network, used to detect foreign matter mixed in tobacco shreds on a tobacco processing production line, characterized in that, include: RGB and near-infrared spectral images of tobacco shreds on the tobacco processing conveyor belt are acquired simultaneously using an industrial camera. RGB images and near-infrared spectral images are respectively input into the RGB branch and near-infrared branch of the dynamic channel reconstruction encoder. The RGB branch uses an asymmetric channel splitting strategy to compress and reconstruct the RGB image feature map of the tobacco, while the near-infrared branch uses a multi-band splitting strategy to perform multi-band weighted processing on the near-infrared spectral image feature map of the tobacco. A temperature sensing sparsity strategy for embedded devices used for foreign object detection in tobacco is introduced for dynamic sparsification processing, and RGB encoded features and near-infrared encoded features are output respectively. The RGB encoding features, the near-infrared encoding features, and the vibration spectrum features of the yarn-making conveyor collected by the vibration sensor are input into the multimodal feature dynamic routing module. The three modal features are weighted and fused through a learnable routing matrix, and a fused feature map is output. The fused feature map is input into the lightweight spatiotemporal detection head, and spatial features are extracted sequentially through depthwise separable convolution and temporal modeling is performed through an LSTM network. When historical frames exist, temporal alignment is performed through a motion compensation module for the running motion of the yarn conveyor belt. Finally, the detection result includes the foreign object position coordinate offset, confidence level, and class probability. Based on the foreign object location information in the detection results, the removal device is controlled to remove the foreign object from the tobacco.

2. The foreign matter detection method for filament fabrication based on a multimodal lightweight dynamic network according to claim 1, characterized in that, The asymmetric channel splitting strategy used in the RGB branch is as follows: For the input feature map Where B is the batch size, C is the number of input channels, and H and W are the height and width of the feature map, respectively. The channel dimensions are split according to a dynamic compression ratio α=0.6: ， , where k=|αC|, indicating that the first k channels are retained for subsequent calculations; This indicates that the remaining channels are compressed into mean channels using global average pooling. To output the recombined feature map; right Lightweight residual blocks constructed using depthwise separable convolutions are used for further feature extraction, and channel attention mechanisms are employed to enhance feature representation.

3. The foreign matter detection method for filament fabrication based on a multimodal lightweight dynamic network according to claim 1, characterized in that, The multi-band splitting strategy employed in the near-infrared branch is as follows: The near-infrared spectral image was split into three channels: 900 nm, 1200 nm, and 1500 nm. ; Each band is used for independent feature extraction through 5×5 convolution, and learnable weights are employed. Perform a weighted summation: ， The weighted multi-band features are enhanced sequentially using spectral attention mechanism and spatial enhancement method, and finally integrated by direct stitching.

4. The foreign matter detection method for filament fabrication based on a multimodal lightweight dynamic network according to claim 1, characterized in that, The temperature sensing sparsity strategy is specifically as follows: The sparsity is dynamically adjusted based on the real-time temperature T of the embedded device, using a Bernoulli sampling matrix. Sparsification of encoded features: ,in The weight matrix is ​​a learnable weight matrix; The Bernoulli sampling matrix The generation probability is controlled by the equipment temperature T: , where σ is the sigmoid function, and T is in degrees Celsius; When the device temperature rises, the sparsity automatically increases to reduce the computational load; when the device temperature falls, the sparsity automatically decreases to make full use of computing resources.

5. The foreign matter detection method for filament fabrication based on a multimodal lightweight dynamic network according to claim 1, characterized in that, The processing procedure of the multimodal feature dynamic routing module includes: Initial routing matrix The identity matrix retains the baseline weights of each modality's own features, with no cross-modal interactions initially. These correspond to the RGB mode, near-infrared mode, and vibrational spectrum mode, respectively. Obtain the normalized temperature T, memory usage M, and power consumption P of the embedded processing device, and construct the device state vector. The weight vector α is obtained by mapping the input s to the fully connected layer. The routing matrix is ​​updated by element-wise multiplication: ； Modal projection is performed on the RGB encoded features, near-infrared encoded features, and vibrational spectrum features through 1×1 convolution, yielding... Then, the updated routing matrix is ​​used to perform weighted fusion of the features of each modality: .

6. The foreign matter detection method for filament fabrication based on a multimodal lightweight dynamic network according to claim 1, characterized in that, The processing steps of the lightweight spatiotemporal detection head include: The fused feature map is processed using depthwise separable convolution. First, spatial features are extracted by channel-wise convolution in channel-grouped groups. Then, channel information is fused by 1×1 pointwise convolution to obtain the spatial features. ; The spatial features Reconstruct the sequence [B, HW, C] and input it into the LSTM network, with its hidden states arranged in sequence. Update, in which and Here, δ represents the weight matrices for the input and hidden states, respectively, and δ is the activation function. When historical frames (prev_frames) exist, inter-frame differential estimation is used to estimate the optical flow offset. Through bilinear sampling To achieve feature alignment, where For sampling grid coordinates; The time series features output by the LSTM network Restored to a spatial format, the 6-dimensional detection output is predicted through 1×1 convolution, including 4 coordinate offsets, 1 confidence score, and 1 class probability.

7. The foreign matter detection method for filament fabrication based on a multimodal lightweight dynamic network according to any one of claims 1 to 6, characterized in that, The detection method further includes: during the inference deployment phase, performing hardware adaptive model compression on the trained model, wherein the hardware adaptive model compression specifically includes: Calculate the importance score of each channel of the input feature map. ; According to the preset pruning ratio Calculate the quantile threshold = Generate a binary mask mask = I(importance > threshold), and set the channels with importance scores below the threshold to zero; Calculate scaling factor Where b is the number of quantization bits, discretizing continuous values ​​into... Each level is operated according to rounding rules. This is to achieve quantitative compression.

8. The foreign matter detection method for filament fabrication based on a multimodal lightweight dynamic network according to claim 7, characterized in that, Different pruning ratios and quantization bit parameters are preset for different target embedded chip architectures.

Images