Tobacco foreign matter segmentation detection method based on bimodal characteristics

Abstract

The invention discloses a tobacco foreign matter segmentation detection method based on bimodal features, and relates to the technical field of tobacco detection, RGB images and spectral images are mixed to serve as input, features are extracted through double encoders, and due to the fact that the spectral images of different substances are obviously different, the problem that foreign matter and tobacco are similar in color is solved; in order to better utilize the features of the two modes, feature fusion is carried out on the different input modes; and then fusing the shallow features and the deep features in the modal fusion result again by using a multi-scale fusion mechanism. And finally, through decoding processing and edge refining processing, a foreign matter segmentation detection range in the tobacco material is accurately positioned. The method effectively relieves the problem that the foreign matter detection effect is poor when the color of the foreign matter is similar to the color of tobacco leaves in tobacco foreign matter detection.

Description

Technical Field

[0001] The present invention relates to the technical field of tobacco detection, and in particular to a tobacco foreign body segmentation detection method based on dual-modal features. Background Art

[0002] Tobacco leaves are the basic raw materials of the tobacco industry, and the quality of tobacco leaves plays a key role in the production quality and operating efficiency of the tobacco industry. Therefore, it is of positive significance to improve the intelligent identification and removal technology of tobacco leaf foreign bodies in the tobacco production process.

[0003] Due to the large number of tobacco leaves, if the manual method is used to sort the debris and foreign matter on the production line, the workload is huge and the implementation process is not realistic. Traditional methods for removing foreign matter from tobacco production lines mainly include laser sorting technology, ECO tracking algorithm, etc., but the reliability of using traditional methods to remove foreign matter from tobacco production lines is not good, the recognition level is not high, and the production efficiency of the production line cannot be guaranteed. With the development of machine vision technology, the research on the design method of foreign matter removal system for tobacco production lines based on machine vision has received great attention. Generally, a CCD camera is used to capture the detection image and convert it into a digital signal, and then the image digital signal is processed by advanced computer hardware and software technology to obtain the required various target image feature values, thereby realizing template matching, pattern recognition, coordinate calculation, grayscale distribution map, etc. Then, according to the results, the image is displayed, the data is output, instructions are issued, and the actuator (including rejection mechanism, robot arm, large area display, alarm device, etc.) is automatically completed. Position adjustment, good and bad screening, data statistics and other automated processes. Compared with traditional methods, the biggest advantages of machine vision are intelligence, precision, speed, reliability, and digitization.

[0004] However, the machine vision-based method is only effective when the color difference between the foreign matter and the tobacco leaves is large. For example, the rejection rate of wood chips, chicken feathers, paper pieces, blue belts, green belts, yellow belts, transparent plastics, etc. is relatively high (reaching more than 90%). However, the colors of some typical foreign matters (such as carton boards, hemp ropes, and black plastics) are similar to or even the same as the color of tobacco leaves, which brings great difficulties to identification. The rejection rate of carton boards and hemp ropes using the original method can only be around 10%, which is difficult to meet production needs. Summary of the invention

[0005] In view of the above, the present invention aims to provide a tobacco foreign body segmentation detection method based on dual-modal features, so as to improve the accuracy of the tobacco foreign body segmentation detection model.

[0006] The technical solution adopted by the present invention is as follows:

[0007] The present invention provides a tobacco foreign body segmentation detection method based on dual-modal features, which includes:

[0008] Collecting the RGB image and the corresponding spectral image of the tobacco material, and inputting them into the pre-built dual-channel encoder for feature extraction respectively;

[0009] Perform modal fusion on the RGB image features and spectral image features of tobacco materials;

[0010] The modal fusion results are then subjected to multi-scale fusion to obtain the final fusion feature map;

[0011] Inputting the final fused feature map into a pre-built decoder to obtain a decoded feature map;

[0012] Based on the decoded feature map, the edges of foreign objects in the tobacco material are refined to complete the segmentation and detection of foreign objects.

[0013] In at least one possible implementation, the dual-path encoder is constructed in the following manner:

[0014] Collect RGB images and spectral images of tobacco materials, annotate foreign objects in the two modal images, and divide them into training sets and test sets;

[0015] A dual-path encoder is constructed based on the training set and the test set, and the U-Net architecture; wherein the dual-path encoders respectively use ResNet18 networks, and each encoder includes several residual blocks.

[0016] In at least one possible implementation, the performing modal fusion includes: setting a modal fusion module corresponding to the number of residual blocks of each encoder;

[0017] The modal fusion block is used to fuse the encoding results output by the residual blocks of the same layer to obtain a modal fusion feature map of each encoding stage.

[0018] In at least one possible implementation manner, obtaining the modality fusion feature map includes:

[0019] Perform adaptive weighted calculation on the two modal features extracted respectively to obtain the first fusion feature;

[0020] Perform cross-attention calculation on the two modal features extracted respectively to obtain the second fusion feature;

[0021] The first fusion feature and the second fusion feature are concatenated and fused and the number of channels are aligned to obtain a modal fusion feature map.

[0022] In at least one possible implementation, the decoder is constructed in the following manner: a number of decoding blocks are set corresponding to the number of modal fusion modules, and each decoding block includes at least one upsampling layer.

[0023] In at least one possible implementation, the multi-scale fusion includes:

[0024] Align the dimensions of the modal fusion features at each encoding stage;

[0025] The modal fusion features after dimension alignment are concatenated and added respectively to obtain concatenated features and added features respectively;

[0026] The concatenated features and the added features are weighted according to preset weights to obtain a multi-scale fusion feature map.

[0027] In at least one possible implementation, thinning the edge of the foreign body includes:

[0028] Perform edge enhancement on the decoded feature map to obtain a target edge enhanced feature map;

[0029] Perform context information enhancement on the target edge enhancement feature map to obtain an edge probability map;

[0030] The edge probability map is binarized and the edges of foreign matter in the tobacco material are determined using a predetermined threshold.

[0031] Compared with the prior art, the main design concept of the present invention is to use a mixture of RGB images and spectral images as input and use dual encoders to extract features respectively. Since the spectral images of different substances are obviously different, it is helpful to solve the problem of similar color between foreign matter and tobacco. In order to better utilize the features of the two modes, the features of the above different input modes are then fused; then, the shallow features and deep features in the modal fusion results are fused again using a multi-scale fusion mechanism. Finally, after decoding processing and edge refinement processing, the foreign matter segmentation detection range in the tobacco material is accurately located. The present invention effectively alleviates the problem of poor foreign matter detection in tobacco foreign matter detection when the foreign matter and tobacco leaves are similar in color. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described below with reference to the accompanying drawings, in which:

[0033] Figure 1 A schematic diagram of a flow chart of a tobacco foreign body segmentation and detection method based on dual-modal features provided in an embodiment of the present invention;

[0034] Figure 2A schematic diagram of an algorithm framework of a tobacco foreign body segmentation detection method based on dual-modal features provided in an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of a modality fusion algorithm provided by an embodiment of the present invention

[0036] Figure 4 A schematic diagram of a multi-scale fusion algorithm provided by an embodiment of the present invention;

[0037] Figure 5 A schematic diagram of an edge thinning algorithm provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0039] In view of the problems mentioned above, the present invention further considers that only using RGB images for machine vision-based tobacco foreign body detection is not effective when the color of the foreign body and the tobacco leaves are similar. Therefore, it is considered to use RGB images and spectral images as inputs in a mixed manner and use dual encoders to extract features respectively in the embodiments provided by the present invention. The spectral images of different substances are obviously different, which is conducive to solving the problem of similar colors between foreign bodies and tobacco. In order to better utilize the features of the two modalities, the present invention also designs a modal fusion module in the embodiment, and realizes the effective expression of different modal features through adaptive weighting and cross-attention. In addition, in order to make full use of shallow features and deep features, a multi-scale fusion module is proposed in the present invention; finally, based on the design of the edge refinement module, the foreign body segmentation detection range can be more accurately located.

[0040] Specifically, the present invention proposes an embodiment of a tobacco foreign body segmentation detection method based on dual-modal features. Figure 1 and Figure 2 shown, including:

[0041] Step S1, collecting the RGB image and the corresponding spectral image of the tobacco material, and inputting them into a pre-built dual-channel encoder respectively, and performing feature extraction respectively;

[0042] Specifically, the RGB image and spectral image of the tobacco raw material are taken as input together, and a dual-channel encoder (dual-stream decoder) is used to extract features of the two respectively. Since the spectral images of different substances are obviously different, after the RGB image is fused with the spectral image, it is helpful to solve the problem of similar color between foreign matter and tobacco.

[0043] The construction method of the dual-path encoder can be referred to as follows:

[0044] First, collect RGB images and spectral images during the process, and annotate foreign objects in the images to complete the production of data labels and divide them into training sets and test sets;

[0045] Secondly, the overall encoding algorithm adopts the U-Net architecture. Both encoders can use the ResNet18 network. Each encoder can also specifically include four residual blocks ( Figure 2 The four residual blocks generate modality fusion feature maps of each stage with dimensions (number of channels × length × width) of 64×56×56, 128×28×28, 256×14×14, and 512×7×7, respectively.

[0046] Step S2, performing modal fusion on the RGB image features and the spectral image features of the tobacco material;

[0047] Recombination Figure 3 As shown, in order to better utilize the features of the two modalities of RGB and spectrum, in some embodiments, the present invention designs a modal fusion module corresponding to the number of residual blocks of each encoder, and realizes the effective expression of different modal features through the mechanism of adaptive weighting and cross attention.

[0048] Specifically, we first transform the two modal features ( , ) is adaptively weighted to enhance the effective feature expression, where Fr represents the image feature and Fs represents the spectral feature. The first fusion feature after preliminary fusion is defined as , the process is as follows:

[0049]

[0050]

[0051]

[0052]

[0053] in, and They correspond to the intermediate features of the two modal features during the fusion process.

[0054] Secondly, the two modal features ( , ) to perform cross attention, so that the two modal features are dynamically interactively fused, and the resulting second fusion feature is defined as , the process is as follows:

[0055]

[0056] Finally, The final modality fusion feature is generated by concatenating and further fusion through 1×1 convolution and aligning the number of channels. , i represents the number of fusion stages, and the process is as follows:

[0057]

[0058] Step S3, performing multi-scale fusion on the modal fusion result to obtain the final fusion feature map;

[0059] For example Figure 4 As shown, in order to make full use of the shallow information and deep information in the fusion features, the present invention also designs a multi-scale fusion module.

[0060] First, the features from the above four modality fusion stages are combined using average pooling and 1×1 convolution operations ( ) to align the dimensions and obtain the corresponding feature expression after alignment ( ), and then concatenate and add the dimensionally aligned features to obtain the corresponding concatenated features , Additive Features , the process is as follows:

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067] Then , Further weighting is performed to highlight effective features, suppress irrelevant features, and produce the final multi-scale fusion feature map Input to the decoder, the process is as follows:

[0068]

[0069]

[0070]

[0071] Step S4, inputting the final fused feature map into a pre-built decoder to obtain a decoded feature map;

[0072] In actual operation, the decoder can also be constructed to correspond to the number of modal fusion modules (that is, the number of residual blocks in each encoder), and set a number of decoding blocks, and each decoding block can also include an upsampling layer (which can be implemented using transposed convolution) and two convolution layers, so that after each upsampling, the size of the feature map will double. This concept can also be expanded to say that, preferably, through a jump connection method, the modal fusion feature map of each stage of the residual blocks corresponding to the two modalities in the two encoders after modal fusion is cascaded (concat) with the feature map of the corresponding encoding blocks after upsampling.

[0073] Step S5: refine the edges of foreign matter in the tobacco material based on the decoded feature graph to complete the segmentation and detection of foreign matter.

[0074] refer to Figure 5 As shown, in order to accurately locate the segmentation and detection range of foreign matter in tobacco materials, an edge refinement module is designed in the present invention.

[0075] First, edge enhancement is performed. The Canny operator is used to enhance the edge pattern of the decoded feature map G output by the decoder, and then L2 normalization is used to increase its generalization ability and reduce calculation errors. Finally, the Hadamard product is performed on the original decoded feature map and the edge-enhanced and normalized intermediate feature map to obtain the target edge-enhanced feature map GE;

[0076] Then, the context of the target edge enhancement feature map is enhanced. The global features Gg and local features Gl of GE are extracted respectively. The local information is extracted using ResBlock, and GE is reshaped to match the size of the local features. When extracting global information, two 1 × 1 convolutions are first applied to the reshaped feature map to obtain an adjacency matrix that stores contextual information. The first convolution layer compresses the feature dimension, and the second convolution layer transfers the N-dimensional features to an N*N adjacency matrix, and L2 normalizes each row of the matrix. This adjacency matrix is ​​used to complete graph reasoning to obtain the global feature map Gg. In order to match the scale of the local features, Gg is reshaped and then connected to Gl. After 1*1 convolution and sigmoid, the edge probability map Gp is ​​obtained;

[0077] Finally, the marginal probability Figure 2 The edge is determined by using a predetermined fixed threshold value to obtain the refined edge of foreign matter in the tobacco material.

[0078] In summary, the mixed use of RGB images and spectral images as input and the use of dual encoders to extract features respectively, due to the obvious differences in the spectral images of different substances, is conducive to solving the problem of similar colors between foreign objects and tobacco; in order to better utilize the features of the two modalities, the features of the above-mentioned different input modalities are then fused; after that, the shallow features and deep features in the modal fusion results are fused again using a multi-scale fusion mechanism. Finally, after decoding processing and edge refinement processing, the foreign object segmentation detection range in the tobacco material is accurately located. The present invention effectively alleviates the problem of poor foreign object detection in tobacco foreign object detection when the foreign object is similar in color to the tobacco leaf.

[0079] If the expressions expressing orientation are mentioned in the embodiments of the present invention, they are relative concepts based on the embodiments. In addition, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, wherein a, b, c can be single or multiple.

[0080] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings, but the above is only a preferred embodiment of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched into a variety of equivalent schemes by those skilled in the art without departing from or changing the design ideas and technical effects of the present invention; therefore, the present invention is not limited to the scope of implementation shown in the drawings, and all changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the specification and drawings, should be within the protection scope of the present invention.

Claims

1. A tobacco foreign body segmentation detection method based on dual-modal features, characterized in that: include: Collecting the RGB image and the corresponding spectral image of the tobacco material, and inputting them into the pre-built dual-channel encoder for feature extraction respectively; Perform modal fusion on the RGB image features and spectral image features of tobacco materials; The modal fusion results are then subjected to multi-scale fusion to obtain the final fusion feature map; Inputting the final fused feature map into a pre-built decoder to obtain a decoded feature map; Based on the decoded feature map, the edges of foreign objects in the tobacco material are refined to complete the segmentation and detection of foreign objects.

2. The tobacco foreign body segmentation detection method based on dual-modal features according to claim 1, characterized in that: The dual-path encoder is constructed in the following manner: Collect RGB images and spectral images of tobacco materials, annotate foreign objects in the two modal images, and divide them into training sets and test sets; A dual-path encoder is constructed based on the training set and the test set, and the U-Net architecture; wherein the dual-path encoders respectively use ResNet18 networks, and each encoder includes several residual blocks.

3. The tobacco foreign body segmentation detection method based on dual-modal features according to claim 2 is characterized in that: The performing of modal fusion includes: setting a modal fusion module corresponding to the number of residual blocks of each encoder; The modal fusion block is used to fuse the encoding results output by the residual blocks of the same layer to obtain a modal fusion feature map of each encoding stage.

4. The tobacco foreign body segmentation detection method based on dual-modal features according to claim 3 is characterized in that: The obtaining of the modality fusion feature map comprises: Perform adaptive weighted calculation on the two modal features extracted respectively to obtain the first fusion feature; Perform cross-attention calculation on the two modal features extracted respectively to obtain the second fusion feature; The first fusion feature and the second fusion feature are concatenated and fused and the number of channels are aligned to obtain a modal fusion feature map.

5. The tobacco foreign body segmentation detection method based on dual-modal features according to claim 3, characterized in that: The construction method of the decoder includes: setting a number of decoding blocks corresponding to the number of modal fusion modules, and each decoding block includes at least one upsampling layer.

6. The tobacco foreign body segmentation detection method based on dual-modal features according to claim 1, characterized in that: The multi-scale fusion includes: Align the dimensions of the modal fusion features at each encoding stage; The modal fusion features after dimension alignment are concatenated and added respectively to obtain concatenated features and added features respectively; The concatenated features and the added features are weighted according to preset weights to obtain a multi-scale fusion feature map.

7. The tobacco foreign body segmentation detection method based on dual-modal features according to any one of claims 1 to 6, characterized in that: Refinement of foreign body edges includes: Perform edge enhancement on the decoded feature map to obtain a target edge enhanced feature map; Perform context information enhancement on the target edge enhancement feature map to obtain an edge probability map; The edge probability map is binarized and the edges of foreign matter in the tobacco material are determined using a predetermined threshold.

Images