Method for identifying fresh cocoon gender and internal marking dead cocoon by near-infrared multi-modal feature fusion
By using a dual-branch deep learning network that fuses near-infrared multimodal features, the spectral sequence and image texture features of fresh cocoons are automatically extracted, solving the problem of efficient identification of male and female fresh cocoons and dead cocoons with inner prints. This improves the automation and accuracy of cocoon quality sorting and promotes the intelligent development of the sericulture and silk industry.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTHWEST UNIV
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies are insufficient for efficiently and non-destructively identifying the sex of fresh cocoons and the presence of dead cocoons inside. Furthermore, they rely on artificial feature engineering and lack sufficient spectral feature analysis capabilities, resulting in low efficiency in cocoon quality sorting.
A near-infrared multimodal feature fusion method is adopted. Through a dual-branch deep learning network of one-dimensional spectral sequence and two-dimensional spectral image, the spectral sequence and image texture features are automatically extracted and adaptively weighted to achieve high-precision identification of male and female and inner dead cocoons.
It achieves high-precision, non-destructive, and rapid identification of fresh cocoons, improves the automation level and production efficiency of cocoon quality sorting, and promotes the intelligent development of the sericulture and silk industry.
Smart Images
Figure CN122135078A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of silkworm cocoon quality sorting technology, specifically a method for identifying the sex of fresh cocoons and the presence of dead cocoons with inner prints by near-infrared multimodal feature fusion. Background Technology
[0002] The sericulture and silk industry is a traditional pillar industry in my country. The stability and improvement of raw silk quality is the core of the industry's development, and the precise selection of raw cocoons is a crucial prerequisite for ensuring raw silk quality. Research and practice show that implementing a differentiated silk reeling process based on male-female identification can optimize process parameters according to the differences in structure and physicochemical properties of male and female cocoons, thereby effectively improving the uniformity and added value of raw silk. However, because the appearance of silkworm cocoons cannot reliably distinguish between male and female, this process is difficult to promote and apply in actual production. Furthermore, dead cocoons with internally printed marks contaminate the cocoon layer due to the decay of the pupal body inside, leading not only to the deterioration of their own silk quality but also to contaminating surrounding normal cocoons during the boiling process, seriously affecting the final quality of raw silk. Therefore, developing a technology that can simultaneously, non-destructively, and efficiently identify male and female marks and dead cocoons with internally printed marks in fresh cocoons is of urgent practical significance for controlling raw material quality from the source and improving the efficiency of the entire industry chain.
[0003] Near-infrared spectroscopy, with its non-destructive, rapid, and sensitive properties to organic components, is considered an ideal method for achieving the aforementioned detection. Current research primarily employs traditional chemometric models such as partial least squares discriminant analysis for sex determination. However, the subtle differences in the internal chemical composition of male and female silkworm cocoons lead to highly similar and nonlinearly overlapping near-infrared spectral characteristics, posing fundamental limitations to traditional methods: firstly, they heavily rely on expert experience for manual feature band selection, a subjective and cumbersome process; secondly, these shallow learning models have limited representational capabilities, making it difficult to autonomously extract deep, abstract nonlinear discriminative patterns from raw high-dimensional spectral data. Therefore, when dealing with fine-grained, highly confusing classification problems like male and female cocoons, model performance easily reaches a bottleneck.
[0004] In recent years, deep learning techniques (such as convolutional neural networks) have provided new approaches for end-to-end automatic extraction of spectral features, enabling the direct learning of hierarchical features from data and enhancing the ability to analyze complex patterns.
[0005] However, in the identification of fresh cocoons, existing deep learning-based methods are mostly limited to a single one-dimensional spectral sequence input. Essentially, they have not broken through the framework of a single data perspective and have failed to fully explore and utilize the complementary discriminative information in spectral data that may be expressed through other dimensions (such as time-domain / frequency-domain structural features).
[0006] Therefore, a new solution is needed to address the above problems. Summary of the Invention
[0007] The purpose of this invention is to provide a method for identifying the sex of fresh cocoons and the presence of dead cocoons with inner prints by near-infrared multimodal feature fusion. This method can automatically and deeply mine complementary information in spectral sequences and image textures, and achieve a high-precision, non-destructive, and rapid identification solution to meet the needs of intelligent sorting of raw cocoons in actual production. This method solves the problems of existing technologies for identifying the sex of fresh cocoons and the presence of dead cocoons with inner prints, such as reliance on manual feature engineering and insufficient ability to resolve fine-grained spectral differences.
[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for identifying the sex of fresh cocoons and the presence of dead cocoons with inner imprints based on near-infrared multimodal feature fusion, comprising at least the following steps:
[0009] S1: Sample Preparation and Spectral Acquisition: A standard batch of silkworms was raised and fresh cocoons were harvested. Double cocoons and obviously secondary cocoons were removed, and the cocoon shells were peeled off. Then, the raw spectra were acquired using a Fourier transform near-infrared spectrometer in diffuse transmission mode. The spectral wavenumber range was [missing information]. Each spectrum contains 922 data points;
[0010] S2: Data preprocessing and dual-modal input construction: The original spectrum is vector normalized to obtain a one-dimensional spectral sequence; then, the recursive graph algorithm is used to convert the sequence into a two-dimensional recursive graph, thereby constructing the image modal representation of the spectrum.
[0011] S3: Dual-branch deep feature extraction: The one-dimensional spectral sequence is input to the first feature extraction branch, and the two-dimensional spectral image is input to the second feature extraction branch; the first feature extraction branch is a feature extraction network based on a one-dimensional convolutional neural network, used to extract local details and channel-position association features of the spectral sequence; the second feature extraction branch is a feature extraction network based on a two-dimensional convolutional neural network, used to extract the texture structure and spatial hierarchy features of the spectral image;
[0012] S4: Adaptive weighted feature fusion: Adaptive average pooling and flattening are performed on the features output from the first feature extraction branch and the second feature extraction branch; then, multimodal fusion is performed using an adaptive weighted fusion strategy to obtain the fused feature vector;
[0013] S5: Classification and Recognition: Input the fused feature vector into the classifier and output the fresh cocoon prediction result. The prediction result is the probability that the identified fresh cocoon belongs to the category of female cocoon, male cocoon, or cocoon with inner dead print.
[0014] Furthermore, S2 includes at least the following steps:
[0015] S2.1: For the original spectral vector Perform vector normalization, then calculate L of vector X. 2 Norm, ,in Let the dimension be the vector. For indexing; then use Values of vectors Normalization , where X norm This represents the normalized vector, which eliminates the dimensional differences between spectral sequences, giving them a uniform scale and resulting in a one-dimensional spectral sequence.
[0016] S2.2: Reconstruct the phase space vector from the one-dimensional spectral sequence using the time-delay embedding method;
[0017]
[0018] in, To construct a vector Specific data, For embedded dimensions, For time delay;
[0019] S2.3: Calculate the Euclidean distance between all pairs of phase space vectors. This forms a distance matrix;
[0020] S2.4: Perform thresholding binary processing on the distance matrix to generate a recursive matrix, which serves as a two-dimensional spectral image. ;
[0021]
[0022] in It is a unit step function. This represents the threshold.
[0023] Furthermore, the first feature extraction branch includes a one-dimensional spatial grouping enhancement module, a one-dimensional coordinate attention module, and a ConvNeXtV2 module;
[0024] The one-dimensional spatial grouping enhancement module is used to focus on key spectral regions, suppress redundant information, and enhance the correlation between local details and the global context. The one-dimensional spatial grouping enhancement module includes channel grouping, global feature calculation, similarity calculation, attention mask generation, and feature weighting.
[0025] The one-dimensional coordinate attention module is used to capture the position dependence and channel association of the spectral sequence and accurately locate discriminative spectral segments. The one-dimensional coordinate attention module includes dimension splitting, feature fusion and transformation, attention generation and feature weighting.
[0026] The ConvNeXtV2 module is used to deeply refine features, reduce channel redundancy, and enhance feature diversity and expressive power. The ConvNeXtV2 module includes depthwise separable convolution, group normalization, channel expansion and activation, global response normalization, and channel compression and residual connection.
[0027] Furthermore, the application process of the first feature extraction branch includes at least the following steps:
[0028] S3.1.1: Input one-dimensional spectral sequence A one-dimensional convolutional layer is used, with a kernel size of 11, a stride of 1, and padding of 5. The number of channels is expanded from 1 to 8, and a feature map with a size of 8×922 is output, which enhances feature representation while preserving the original resolution.
[0029] S3.1.2: Introducing a one-dimensional spatial grouping enhancement module, the input feature map is divided into G groups along the channel dimension. The similarity between the spatial location features in each group and the global average features is calculated. After normalization and learnable scaling, an attention mask is generated to focus on key spectral regions.
[0030] S3.1.3: Six sequentially connected downsampling stages are used. Each stage includes an average pooling layer with a kernel size of 2 and a one-dimensional convolutional layer. The kernel sizes of the one-dimensional convolutional layers in the six stages are [11, 11, 9, 7, 7, 5] and the number of output channels are [16, 16, 32, 32, 64, 64] respectively. At this time, the feature map size is changed from 922×8 to 14×64.
[0031] S3.1.4: A one-dimensional coordinate attention module is introduced on the 4th to 7th feature maps in the 6 downsampling stages. The one-dimensional coordinate attention module is a one-dimensional CA module. The one-dimensional coordinate attention module performs one-dimensional global pooling along the sequence direction to encode position information and generates an attention map that combines channel and position weights through feature transformation to accurately locate discriminative spectral segments.
[0032] S3.1.5: Deploy the ConvNeXtV2 module on all feature maps of the first feature extraction branch. The ConvNeXtV2 module uses large kernel depthwise separable convolution and global response normalization to effectively reduce channel redundancy and enhance feature diversity.
[0033] Furthermore, the second feature extraction branch includes a shuffling attention network, a coordinate attention mechanism, and a residual network module based on partial convolution;
[0034] The shuffling attention network module is used to apply channel and spatial attention separately. The shuffling of channels enhances feature interaction, suppresses noise and strengthens key textures. The shuffling attention network module includes feature grouping, feature splitting, channel attention calculation, spatial attention calculation, feature weighting and fusion.
[0035] The two-dimensional coordinate attention module is used to capture the long-range spatial dependence and channel correlation of the image and accurately locate discriminative texture regions. The two-dimensional coordinate attention module includes bidirectional global pooling, feature reshaping and fusion, feature transformation and activation, attention weight generation and feature weighting.
[0036] The partially convolutional residual network module is used to reduce the computational cost while ensuring feature extraction capability and avoid gradient vanishing through residual connections. The partially convolutional residual network module includes a partially convolutional layer 1, batch normalization and activation, a partially convolutional layer 2, batch normalization and residual connections.
[0037] Furthermore, the application process of the second feature extraction branch includes at least the following steps:
[0038] S3.2.1: Input a two-dimensional spectral image ∈R 1×458×458 A two-dimensional convolutional layer with a kernel size of 7×7, a stride of 2, and padding of 3 is used to expand the number of channels from 1 to 20 and output a feature map. At this time, the feature map size is 20×229×229.
[0039] S3.2.2: Introduce a shuffling attention network module, which divides the feature map into two parts after grouping, applies channel attention and spatial attention to them respectively, and finally concatenates and shuffles the channels. Channel attention captures the global context through global average pooling, and spatial attention captures spatial relationships through group normalization. Together, they suppress noise and enhance salient features.
[0040] S3.2.3: Four downsampling stages are adopted, each stage consisting of a 2×2 average pooling layer and a two-dimensional convolutional layer. The kernel size of the two-dimensional convolutional layer in the four stages is [5, 5, 3, 3], and the number of output channels is [40, 60, 80, 100], respectively. The feature map size is correspondingly changed from 20×229×229 to 14×14×100.
[0041] S3.2.4: A two-dimensional coordinate attention module is introduced on the second to fourth feature maps in the four downsampling stages to capture the long-range spatial dependence of important texture regions in the image. The two-dimensional coordinate attention module is the two-dimensional CA module.
[0042] S3.2.5: Deploy a residual network module based on partial convolution on all feature maps of the second feature extraction branch. The partial convolution only performs spatial convolution on 1 / 4 of the input channels, and the remaining channels are identically mapped.
[0043] Furthermore, grouping normalization is applied after each convolutional layer of the first feature extraction branch;
[0044] Batch normalization is applied after each convolutional layer in the second feature extraction branch;
[0045] Both the first feature extraction branch and the second feature extraction branch use the same scaled Tanh activation function.
[0046] Furthermore, the multimodal adaptive weighted fusion strategy includes at least the following steps:
[0047] Adaptive average pooling (compressed to 3×64 and 3×3×100) is performed on the feature maps output by the first feature extraction branch (14×64) and the feature maps output by the second feature extraction branch (14×14×100), respectively, and the pooled features are flattened into the first feature vector F. 1d With the second eigenvector F 2d (First eigenvector F) 1d Flattened to 192 dimensions, the second eigenvector F 2d (900 dimensions)
[0048] Assign corresponding first learnable weights to the first and second feature vectors. With the second learnable weight The first and second learnable weights are normalized using the Softmax function.
[0049]
[0050] in, This is a natural exponential function, used to amplify the differences in the original weights; The learnable weights are the normalized weights of the first feature vector; The learnable weights are the normalized weights of the second feature vector;
[0051] Multiply the normalized first learnable weight by the batch-normalized first feature vector to obtain the weighted first feature;
[0052] The normalized second learnable weight is multiplied by the batch-normalized second feature vector to obtain the weighted second feature;
[0053] The weighted first feature and the weighted second feature are concatenated to obtain the fused feature vector;
[0054]
[0055] in, This represents a vector concatenation operation; ; This represents element-wise multiplication; Indicates to Batch normalization operation; Indicates to Batch normalization operation.
[0056] Furthermore, the classifier includes a first fully connected layer, a Dropout layer, and a second fully connected layer connected in sequence;
[0057] The input to the classifier is ;
[0058] The first fully connected layer is used to reduce the dimension of the input fused feature vector to half (1092-dimensional input, 546-dimensional output).
[0059] The dropout rate of the Dropout layer is set to 50%;
[0060] The output of the second fully connected layer corresponds to the scores of three categories: female cocoon, male cocoon, and dead cocoon with inner imprint (546-dimensional input, 3-dimensional output).
[0061] The classifier converts the score into class probabilities using the Softmax function;
[0062]
[0063] in, c is the natural exponential function; i This represents the output value of the i-th neuron in the final fully connected layer. It represents the total number of categories; It is the second fully connected layer of the classifier. The original output of each neuron, =1, 2, 3 correspond to female cocoons, male cocoons, and cocoons with inner dead printing, respectively.
[0064] Compared with the prior art, the beneficial effects of the present invention are:
[0065] 1. This invention constructs a dual-modal input of a one-dimensional spectral sequence and a two-dimensional recursive image, and uses a dual-branch network to mine their deep features respectively, effectively overcoming the information limitations of a single data perspective and making full use of the complementarity of different modal features; through a learnable adaptive weighted fusion strategy, it dynamically integrates heterogeneous information, significantly improving the ability to distinguish between male and female cocoons with highly overlapping spectral features.
[0066] 2. This invention adopts an end-to-end network architecture, realizing fully automated processing from raw spectra to recognition results, eliminating traditional manual feature engineering, meeting the requirements of online rapid non-destructive testing, and significantly improving production efficiency.
[0067] 3. The network branches of this invention integrate multiple attention mechanisms and deep convolution modules, which enhance the ability to suppress noise and capture subtle differences; combined with training strategies such as data augmentation and transfer learning, it effectively alleviates the tendency of overfitting under limited samples.
[0068] 4. This invention provides a high-precision, high-efficiency, and automated non-destructive sorting solution for fresh cocoons, providing core technical references for the development of intelligent cocoon sorting equipment, optimization of cocoon drying and boiling processes, and upgrading of silk quality. It is expected to promote the development of the sericulture and silk industry towards refinement and intelligence, and has significant application prospects. Attached Figure Description
[0069] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0070] Figure 1 This is a flowchart illustrating the implementation of the present invention;
[0071] Figure 2 The vector-normalized spectrum and the spectrum transformed by the recursive graph are input for this invention;
[0072] Figure 3 This is a structural diagram of the present invention;
[0073] Figure 4 This is a schematic diagram of the one-dimensional spatial grouping enhancement structure of the present invention;
[0074] Figure 5 This is a schematic diagram of the two-dimensional mixed-wash attention network structure of the present invention;
[0075] Figure 6 This is a schematic diagram of the coordinate attention structure of the present invention;
[0076] Figure 7 This is a schematic diagram of the one-dimensional ConvNeXtV2 structure of the present invention;
[0077] Figure 8 This is a schematic diagram of a portion of the convolutional structure of the present invention;
[0078] Figure 9 This is a schematic diagram of the two-dimensional ResNet structure of the present invention. Detailed Implementation
[0079] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0080] This invention proposes a dual-branch feature deep fusion strategy: a convolutional neural network is used to extract heterogeneous deep features in parallel from a one-dimensional near-infrared spectral sequence and its two-dimensional spectral image transformed by a recursive graph algorithm (the former focuses on sequence patterns, while the latter captures texture structures). An adaptive weighted fusion network is then constructed for collaborative discrimination. This method provides a more efficient technical means for accurate, rapid, and non-destructive identification of the sex of fresh cocoons and the presence of dead cocoons with inner markings, contributing to the development of intelligent and automated cocoon quality sorting.
[0081] Specifically:
[0082] Example 1:
[0083] Please see Figure 1 A near-infrared spectral multimodal fusion method for identifying the sex of fresh cocoons and the presence of dead cocoons with inner imprints includes at least the following steps:
[0084] S1: Sample Preparation and Spectral Acquisition: A standard batch of silkworms was raised and fresh cocoons were harvested. Double cocoons and obviously secondary cocoons were removed, and the cocoon shells were peeled off. Then, the raw spectra were acquired using a Fourier transform near-infrared spectrometer in diffuse transmission mode. The spectral wavenumber range was [missing information]. Each spectrum contains 922 data points;
[0085] S2: Data preprocessing and dual-modal input construction: The original spectrum is vector normalized to obtain a one-dimensional spectral sequence; then, the recursive graph algorithm is used to convert the sequence into a two-dimensional recursive graph, thereby constructing the image modal representation of the spectrum.
[0086] S3: Dual-branch deep feature extraction: The one-dimensional spectral sequence is input to the first feature extraction branch, and the two-dimensional spectral image is input to the second feature extraction branch; the first feature extraction branch is a feature extraction network based on a one-dimensional convolutional neural network, used to extract local details and channel-position association features of the spectral sequence; the second feature extraction branch is a feature extraction network based on a two-dimensional convolutional neural network, used to extract the texture structure and spatial hierarchy features of the spectral image;
[0087] S4: Adaptive weighted feature fusion: Adaptive average pooling and flattening are performed on the features output from the first feature extraction branch and the second feature extraction branch; then, multimodal fusion is performed using an adaptive weighted fusion strategy to obtain the fused feature vector;
[0088] S5: Classification and Recognition: Input the fused feature vector into the classifier and output the fresh cocoon prediction result. The prediction result is the probability that the identified fresh cocoon belongs to the category of female cocoon, male cocoon, or cocoon with inner dead print.
[0089] S2 includes at least the following steps:
[0090] S2.1: For the original spectral vector Perform vector normalization, then calculate L of vector X. 2 Norm, ,in Let the dimension be the vector. For indexing; then use Values of vectors Normalization , where X norm This represents the normalized vector, which eliminates the dimensional differences between spectral sequences, giving them a uniform scale and resulting in a one-dimensional spectral sequence.
[0091] S2.2: Reconstruct the phase space vector from the one-dimensional spectral sequence using the time-delay embedding method;
[0092]
[0093] in, To construct a vector Specific data, For embedded dimensions, For time delay;
[0094] S2.3: Calculate the Euclidean distance between all pairs of phase space vectors. This forms a distance matrix;
[0095] S2.4: Perform thresholding binary processing on the distance matrix to generate a recursive matrix, which serves as a two-dimensional spectral image. ;
[0096]
[0097] in It is a unit step function. This represents the threshold.
[0098] Through the above transformation, one-dimensional spectral sequence features are effectively mapped to spatial texture patterns in a two-dimensional image, thus providing structured input for the subsequent second feature branch.
[0099] The first feature extraction branch includes a one-dimensional spatial grouping enhancement module, a one-dimensional coordinate attention module, and a ConvNeXtV2 module;
[0100] See Figure 4The one-dimensional spatial grouping enhancement module is used to focus on key spectral regions, suppress redundant information, and enhance local details and global correlations. This module includes channel grouping, global feature calculation, similarity calculation, attention mask generation, and feature weighting. In channel grouping, the input feature map (dimension: C×N, where C is the number of channels and N is the length of the spectral sequence) is divided into G groups along the channel dimension, with each group having a dimension of (C / G)×N. In global feature calculation, each group of feature maps is subjected to global average pooling to obtain the global feature vector for each group (dimension: (C / G)×N). In the similarity calculation, the cosine similarity between each spatial feature within each group and the corresponding global feature of the group is calculated to generate a similarity matrix (dimension: 1×N). In the attention mask generation, the similarity matrix is L2 normalized, and the weights are adjusted by a learnable scaling factor (parameter α). After activation by the Sigmoid function, the intra-group attention mask is obtained. In the feature weighting, each group attention mask is multiplied positionally with the original group feature map, and finally the weighted feature maps of all groups are concatenated to output the enhanced feature map (dimension: C×N).
[0101] The one-dimensional coordinate attention module is used to capture the positional dependence and channel association of spectral sequences, accurately locating discriminative spectral segments. The module includes dimensionality splitting, feature fusion and transformation, attention generation, and feature weighting. In dimensionality splitting, the input feature map is divided into positional features along the sequence length. One-dimensional global pooling is performed on the sequence from start to end and from end to start, resulting in two positional feature vectors (each dimension: C×1). In feature fusion and transformation, the two positional feature vectors are concatenated (dimension: C×2), and the number of channels is compressed to C / r (where r is the dimensionality reduction coefficient, taken as r=16 in this invention) through a 1×1 convolutional layer, followed by BatchNorm and scaled Tanh activation functions. In attention generation, another 1×1 convolutional layer restores the number of channels to C, splitting it into two channel weight vectors (each dimension: C×1), corresponding to the two positional directions. In feature weighting, the two channel weight vectors are activated by Sigmoid and multiplied channel-wise and position-wise with the original input feature map to output an enhanced feature map (dimension: C×N).
[0102] See Figure 7 The ConvNeXtV2 module is used to deeply refine features, reduce channel redundancy, and enhance feature diversity and expressive power. The ConvNeXtV2 module includes depthwise separable convolution, group normalization, channel expansion and activation, global response normalization, and channel compression and residual connection.
[0103] In the depthwise separable convolution, a one-dimensional depthwise separable convolution with a kernel size of 7 (number of groups = C) is applied to the input feature map (C×N) while keeping the dimension unchanged (C×N) to capture long program column dependencies. In GroupNorm, the convolution output features are grouped and normalized to stabilize the training process and improve generalization ability. In Channel Expansion and Activation, the number of channels is expanded to 4C through a 1×1 convolutional layer, and nonlinearity is introduced by scaling the Tanh activation function. In Global Response Normalization (GRN), the feature map is globally L2 normalized to enhance the discriminativeness of the features. In Channel Compression and Residual Connection, the number of channels is compressed back to C through a 1×1 convolutional layer, and LayerScale (learnable scalar weights) is added to adjust the feature intensity. Finally, a residual connection (F(X)+X) is performed with the original input feature map to output a refined feature map (dimension: C×N).
[0104] The application process of the first feature extraction branch includes at least the following steps:
[0105] S3.1.1: Input one-dimensional spectral sequence A one-dimensional convolutional layer is used, with a kernel size of 11, a stride of 1, and padding of 5. The number of channels is expanded from 1 to 8, and a feature map with a size of 8×922 is output, which enhances feature representation while preserving the original resolution.
[0106] S3.1.2: Introducing a one-dimensional spatial grouping enhancement module, the input feature map is divided into G groups along the channel dimension. The similarity between the spatial location features in each group and the global average features is calculated. After normalization and learnable scaling, an attention mask is generated to focus on key spectral regions.
[0107] S3.1.3: Six sequentially connected downsampling stages are used (the six downsampling stages correspond to seven feature maps, which include the feature map after the initial input enhancement and the feature map output by each downsampling stage). Each stage includes an average pooling layer with a kernel size of 2 and a one-dimensional convolutional layer. The kernel size of the one-dimensional convolutional layer in the six stages is [11, 11, 9, 7, 7, 5], and the number of output channels is [16, 16, 32, 32, 64, 64], respectively. At this time, the feature map size is changed from 922×8 to 14×64.
[0108] S3.1.4: A one-dimensional coordinate attention module is introduced on the 4th to 7th feature maps in the 6 downsampling stages. The one-dimensional coordinate attention module is also known as the one-dimensional CA module. The one-dimensional coordinate attention module performs one-dimensional global pooling along the sequence direction to encode position information and generates an attention map that combines channel and position weights through feature transformation to accurately locate discriminative spectral segments.
[0109] S3.1.5: Deploy the ConvNeXtV2 module on all feature maps of the first feature extraction branch. The ConvNeXtV2 module uses large kernel depthwise separable convolution and global response normalization to effectively reduce channel redundancy and enhance feature diversity.
[0110] The second feature extraction branch includes a shuffling attention network, a coordinate attention mechanism, and a residual network module based on partial convolution;
[0111] See Figure 5 The shuffling attention network module is used to apply channel and spatial attention separately. Shuffling channels enhances feature interactions, suppresses noise, and strengthens key textures. The shuffling attention network module includes feature grouping, feature splitting, channel attention calculation, spatial attention calculation, feature weighting and fusion, and...
[0112] In feature grouping, the input feature map (dimension: C×H×W, where C is the number of channels and H / W is the image height / width) is divided into G groups along the channel dimension, with each group having a dimension of (C / G)×H×W. In feature splitting, each group of features is further split into two sub-feature maps (dimensions: (C / (2G))×H×W), which are used for channel attention and spatial attention calculations, respectively. In channel attention calculation, the first sub-feature map is subjected to global average pooling to obtain the global channel features within the group (dimension: (C / (2G))×1×1), which are then processed by a 1×1 convolutional layer (F(x)=Wx+b) and activated by a Sigmoid layer to generate channels. Attention weights are calculated as follows: In spatial attention calculation, group normalization (GN) is performed on the second sub-feature map to capture local spatial relationships. After passing through a 1×1 convolutional layer and sigmoid activation, spatial attention weights are generated. In feature weighting and fusion, channel attention weights are multiplied with the first sub-feature map channel by channel, and spatial attention weights are multiplied with the second sub-feature map position by position. The two weighted features are then concatenated (dimension: (C / G)×H×W). In channel shuffling, channel dimension shuffling is performed on the fused feature maps of all groups to break the channel isolation between groups and output the enhanced feature map (dimension: C×H×W).
[0113] See Figure 6 The two-dimensional coordinate attention module is used to capture the long-range spatial dependence and channel correlation of the image and accurately locate discriminative texture regions. The two-dimensional coordinate attention module includes bidirectional global pooling, feature reshaping and fusion, feature transformation and activation, attention weight generation and feature weighting.
[0114] Bidirectional global pooling involves performing one-dimensional global average pooling on the input feature map (C×H×W) along the height (H) and width (W) directions respectively, resulting in positional feature vectors in two directions—pooling along H outputs (C×1×W), and pooling along W outputs (C×H×1). In feature reshaping and fusion, the positional feature vectors in the two directions are reshaped into one-dimensional vectors of (C×(H+W)), and concatenated to obtain the fused positional feature (C×2). In feature transformation and activation, a 1×1 convolutional layer compresses the number of fused feature channels to C / r (r is the dimensionality reduction coefficient; in this invention, r=16). The BatchNorm and scaled Tanh activation functions are used to introduce non-linearity to enhance feature representation. Attention weights are generated by restoring the number of channels to C through another 1×1 convolutional layer, splitting them into two channel weight vectors (each dimension: C×1), corresponding to the height and width directions respectively. After Sigmoid activation, attention weights in the two directions are obtained. Feature weighting is achieved by multiplying the attention weights in the height direction with the original feature map at each height position, and multiplying the attention weights in the width direction with the original feature map at each width position. The two weighting results are superimposed to output an enhanced feature map (C×H×W).
[0115] See Figure 7 and Figure 9 The partially convolutional residual network module is used to reduce the computational cost while ensuring feature extraction capability. Gradient vanishing is avoided through residual connections. The partially convolutional residual network module includes partially convolutional layer 1, batch normalization and activation, partially convolutional layer 2, batch normalization and residual connections.
[0116] Partial convolutional layer 1 applies two-dimensional partial convolution (PConv2d) to the input feature map (C×H×W), performing 3×3 spatial convolution only on 1 / 4 of the input channels, while maintaining the identity mapping for the remaining 3 / 4 channels. The output feature map dimension remains C×H×W. Batch normalization and activation are performed on the partial convolution output using BatchNorm to stabilize training, followed by scaling the Tanh activation function to introduce non-linearity. Partial convolutional layer 2 repeats the partial convolution operation of partial convolutional layer 1 to further refine the features, maintaining the output dimension C×H×W. Batch normalization is performed on the second partial convolution output using BatchNorm. The residual connection adds the processed feature map element-wise to the original input feature map (F(X)+X) to alleviate the gradient vanishing problem in deep networks, outputting a refined feature map (C×H×W).
[0117] The application process of the second feature extraction branch includes at least the following steps:
[0118] S3.2.1: Input a two-dimensional spectral image ∈R 1×458×458A two-dimensional convolutional layer with a kernel size of 7×7, a stride of 2, and padding of 3 is used to expand the number of channels from 1 to 20 and output a feature map. At this time, the feature map size is 20×229×229.
[0119] S3.2.2: Introducing a shuffling attention network module, the feature map is grouped and divided into two parts, and channel attention and spatial attention are applied to them respectively. Finally, the channels are concatenated and shuffled. Channel attention captures the global context through global average pooling, and spatial attention captures spatial relationships through group normalization. Together, they suppress noise and enhance salient features. The fused features output by the shuffling attention network module are downsampled and compressed in dimensionality. Coordinate attention can further capture long-range spatial dependencies.
[0120] S3.2.3: Four downsampling stages are used, each stage consisting of a 2×2 average pooling layer and a two-dimensional convolutional layer. The kernel size of the two-dimensional convolutional layer in the four stages is [5, 5, 3, 3], and the number of output channels is [40, 60, 80, 100], respectively. The feature map size is changed from 20×229×229 to 14×14×100 accordingly.
[0121] S3.2.4: A two-dimensional coordinate attention module is introduced on the second to fourth feature maps in the four downsampling stages to capture the long-range spatial dependence of important texture regions in the image. The two-dimensional coordinate attention module is also known as the two-dimensional CA module.
[0122] S3.2.5: Deploy a residual network module based on partial convolution on all feature maps of the second feature extraction branch. Partial convolution only performs spatial convolution on 1 / 4 of the input channels, with the remaining channels mapped identically. This results in FLOPs and memory accesses being only about 1 / 16 and 1 / 4 of standard convolution, respectively. Combined with residual connections H(x) = F(x) + x, the redundant computation of partial convolution is skipped, directly transmitting the original texture features. Simultaneously, the detailed textures learned by convolution are superimposed, balancing computational efficiency and feature representation capability, significantly improving efficiency while maintaining performance.
[0123] Grouped normalization is applied after each convolutional layer in the first feature extraction branch;
[0124] Batch normalization is applied after each convolutional layer in the second feature extraction branch;
[0125] Both the first and second feature extraction branches use the same scaled Tanh activation function.
[0126] The multimodal adaptive weighted fusion strategy includes at least the following steps:
[0127] Adaptive average pooling (compressed to 3×64 and 3×3×100) is performed on the feature maps output by the first feature extraction branch (14×64) and the feature maps output by the second feature extraction branch (14×14×100), respectively, and the pooled features are flattened into the first feature vector F. 1d With the second eigenvector F 2d (First eigenvector F) 1d Flattened to 192 dimensions, the second eigenvector F 2d (900 dimensions)
[0128] Assign corresponding first learnable weights to the first and second feature vectors. With the second learnable weight The first and second learnable weights are normalized using the Softmax function.
[0129]
[0130] in, This is a natural exponential function, used to amplify the differences in the original weights; The learnable weights are the normalized weights of the first feature vector; The learnable weights are the normalized weights of the second feature vector;
[0131] Multiply the normalized first learnable weight by the batch-normalized first feature vector to obtain the weighted first feature;
[0132] The normalized second learnable weight is multiplied by the batch-normalized second feature vector to obtain the weighted second feature.
[0133] The weighted first feature and the weighted second feature are concatenated to obtain the fused feature vector;
[0134]
[0135] in, This represents a vector concatenation operation; ; This represents element-wise multiplication; Indicates to Batch normalization operation; Indicates to Batch normalization operation.
[0136] The classifier consists of a first fully connected layer, a Dropout layer, and a second fully connected layer connected in sequence.
[0137] The input to the classifier is ;
[0138] The first fully connected layer is used to reduce the dimension of the input fused feature vector to half (1092-dimensional input, 546-dimensional output).
[0139] The dropout rate for the Dropout layer is set to 50%.
[0140] The output of the second fully connected layer corresponds to the scores of three categories: female cocoons, male cocoons, and dead cocoons with inner imprints (546-dimensional input, 3-dimensional output).
[0141] The classifier converts scores into class probabilities using the Softmax function;
[0142]
[0143] in, c is the natural exponential function; i This represents the output value of the i-th neuron in the final fully connected layer. It represents the total number of categories; It is the second fully connected layer of the classifier. The original output of each neuron, =1, 2, 3 correspond to female cocoons, male cocoons, and cocoons with inner dead printing, respectively.
[0144] Example 2:
[0145] Based on the above embodiments, a loss function and optimization method for Embodiment 1 are specifically proposed.
[0146] The prediction error is calculated using the cross-entropy loss function, and the network parameters are updated using the AdamW optimizer.
[0147] Training is performed using the cross-entropy loss function;
[0148]
[0149] in, For true one-hot encoded tags; It is the natural logarithm function;
[0150] The AdamW optimizer, in conjunction with a cosine annealing learning rate scheduler, employs a phased transfer learning strategy to overcome overfitting caused by limited training data.
[0151] Pre-training and freezing: The first and second feature branches are independently optimized through ablation experiments to obtain the optimal feature extractor parameters, which are then frozen during the subsequent training of the complete network.
[0152] End-to-end fine-tuning: After constructing the complete dual-branch network, only the weight parameters (first learnable weights) of the adaptive weighted fusion module are adjusted. With the second learnable weight The parameters of the classifier are used for training.
[0153] Based on the content described in the above embodiments, the following experimental verification is proposed:
[0154] Within the multimodal feature fusion framework, the method of this invention achieves optimal performance. Specifically:
[0155] In the task of identifying the sex of silkworm cocoons, the best traditional feature extraction method (SG smoothing + iPLS spectral feature extraction + PLS-DA classifier) has an accuracy of 92.71%, the best 1D-CNN has an accuracy of 94.79%, the best 2D-CNN has an accuracy of 96.53%, while the extraction method proposed in this invention achieves an accuracy of 97.22%.
[0156] In the task of identifying dead cocoons with internal imprints, the accuracy rates of the four methods mentioned above are 98.76%, 99.55%, 99.55%, and 99.55%, respectively.
[0157] In summary, this invention proposes to perform multimodal fusion of one-dimensional spectral sequences and their derived two-dimensional recursive images, construct a dual-branch neural network to extract their deep features, and dynamically integrate heterogeneous information using an adaptive weighting strategy. This fully utilizes the complementarity between different modalities, significantly improving the identification accuracy of fresh cocoons. Furthermore, this invention employs an end-to-end fully automated processing architecture, achieving high-precision and high-efficiency identification. Through the above design, it overcomes the limitations of single near-infrared spectral data in distinguishing fresh cocoons with highly overlapping features. Moreover, this invention provides a key solution for non-destructive automated sorting of fresh cocoons, effectively supporting the development of intelligent cocoon sorting equipment, optimization of cocoon drying and boiling processes, and upgrading of silk quality, possessing significant industrial application value and economic benefits.
[0158] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A method for identifying the sex of fresh cocoons and the presence of dead cocoons with inner imprints by near-infrared multimodal feature fusion, characterized in that: At least the following steps are included: S1: Sample Preparation and Spectral Acquisition: A standard batch of silkworms was raised and fresh cocoons were harvested. Double cocoons and obviously secondary cocoons were removed, and the cocoon shells were peeled off. Then, the raw spectra were acquired using a Fourier transform near-infrared spectrometer in diffuse transmission mode. The spectral wavenumber range was [missing information]. Each spectrum contains 922 data points; S2: Data preprocessing and dual-modal input construction: The original spectrum is vector normalized to obtain a one-dimensional spectral sequence; then, the recursive graph algorithm is used to convert the sequence into a two-dimensional recursive graph, thereby constructing the image modal representation of the spectrum. S3: Dual-branch deep feature extraction: The one-dimensional spectral sequence is input to the first feature extraction branch, and the two-dimensional spectral image is input to the second feature extraction branch; the first feature extraction branch is a feature extraction network based on a one-dimensional convolutional neural network, used to extract local details and channel-position association features of the spectral sequence; the second feature extraction branch is a feature extraction network based on a two-dimensional convolutional neural network, used to extract the texture structure and spatial hierarchy features of the spectral image; S4: Adaptive weighted feature fusion: Adaptive average pooling and flattening are performed on the features output from the first feature extraction branch and the second feature extraction branch; then, multimodal fusion is performed using an adaptive weighted fusion strategy to obtain the fused feature vector; S5: Classification and Recognition: Input the fused feature vector into the classifier and output the fresh cocoon prediction result. The prediction result is the probability that the identified fresh cocoon belongs to the category of female cocoon, male cocoon, or cocoon with inner dead print.
2. The method for identifying the sex of fresh cocoons and the inner dead cocoons by near-infrared multimodal feature fusion according to claim 1, characterized in that: S2 includes at least the following steps: S2.1: For the original spectral vector Perform vector normalization, then calculate L of vector X. 2 Norm, ,in Let the dimension be the vector. For indexing; then use Values of vectors Normalization , where X norm This represents the normalized vector, which eliminates the dimensional differences between spectral sequences, giving them a uniform scale and resulting in a one-dimensional spectral sequence. S2.2: Reconstruct the phase space vector from the one-dimensional spectral sequence using the time-delay embedding method; in, To construct a vector Specific data, For embedded dimensions, For time delay; S2.3: Calculate the Euclidean distance between all pairs of phase space vectors. This forms a distance matrix; S2.4: Perform thresholding binary processing on the distance matrix to generate a recursive matrix, which serves as a two-dimensional spectral image. ; in It is a unit step function. This represents the threshold.
3. The method for identifying the sex of fresh cocoons and the inner dead cocoons by near-infrared multimodal feature fusion according to claim 1, characterized in that: The first feature extraction branch includes a one-dimensional spatial grouping enhancement module, a one-dimensional coordinate attention module, and a ConvNeXtV2 module; The one-dimensional spatial grouping enhancement module is used to focus on key spectral regions, suppress redundant information, and enhance the correlation between local details and the global context. The one-dimensional spatial grouping enhancement module includes channel grouping, global feature calculation, similarity calculation, attention mask generation, and feature weighting. The one-dimensional coordinate attention module is used to capture the position dependence and channel association of the spectral sequence and accurately locate discriminative spectral segments. The one-dimensional coordinate attention module includes dimension splitting, feature fusion and transformation, attention generation and feature weighting. The ConvNeXtV2 module is used to deeply refine features, reduce channel redundancy, and enhance feature diversity and expressive power. The ConvNeXtV2 module includes depthwise separable convolution, group normalization, channel expansion and activation, global response normalization, and channel compression and residual connection.
4. The method for identifying the sex of fresh cocoons and the inner dead cocoons by near-infrared multimodal feature fusion according to claim 3, characterized in that: The application process of the first feature extraction branch includes at least the following steps: S3.1.1: The input one-dimensional spectral sequence is processed by a one-dimensional convolutional layer to expand the channels and output a feature map; S3.1.2: Introducing a one-dimensional spatial grouping enhancement module, the input feature map is divided into G groups along the channel dimension, and the similarity between each spatial location feature in each group and the global average feature is calculated. After normalization and learnable scaling, an attention mask is generated. S3.1.3: Six sequentially connected downsampling stages are used, each stage including an average pooling layer with a kernel size of 2 and a one-dimensional convolutional layer; S3.1.4: A one-dimensional coordinate attention module is introduced on the 4th to 7th feature maps in the 6 downsampling stages. The one-dimensional coordinate attention module performs one-dimensional global pooling along the sequence direction to encode position information and generates an attention map that combines channel and position weights through feature transformation to accurately locate discriminative spectral segments. S3.1.5: Deploy the ConvNeXtV2 module on all feature maps of the first feature extraction branch.
5. The method for identifying the sex of fresh cocoons and the inner dead cocoons by near-infrared multimodal feature fusion according to claim 4, characterized in that: The second feature extraction branch includes a shuffling attention network, a coordinate attention mechanism, and a residual network module based on partial convolution; The shuffling attention network module is used to apply channel and spatial attention separately. The shuffling of channels enhances feature interaction, suppresses noise and strengthens key textures. The shuffling attention network module includes feature grouping, feature splitting, channel attention calculation, spatial attention calculation, feature weighting and fusion. The two-dimensional coordinate attention module is used to capture the long-range spatial dependence and channel correlation of the image and accurately locate the discriminative texture region. The two-dimensional coordinate attention module includes bidirectional global pooling, feature reshaping and fusion, feature transformation and activation, attention weight generation and feature weighting. The partially convolutional residual network module is used to reduce the computational cost while ensuring feature extraction capability and avoid gradient vanishing through residual connections. The partially convolutional residual network module includes a partially convolutional layer 1, batch normalization and activation, a partially convolutional layer 2, batch normalization and residual connections.
6. The method for identifying the sex of fresh cocoons and the inner dead cocoons by near-infrared multimodal feature fusion according to claim 5, characterized in that: The application process of the second feature extraction branch includes at least the following steps: S3.2.1: Input a two-dimensional spectral image, use a two-dimensional convolutional layer to expand the channels and output a feature map; S3.2.2: Introduce a shuffling attention network module, which divides the feature map into two parts after grouping, applies channel attention and spatial attention to them respectively, and finally concatenates and shuffles the channels. Channel attention captures the global context through global average pooling, and spatial attention captures spatial relationships through group normalization. Together, they suppress noise and enhance salient features. S3.2.3: Four downsampling stages are used, each stage consisting of a 2×2 average pooling layer and a two-dimensional convolutional layer. S3.2.4: A two-dimensional coordinate attention module is introduced on the second to fourth feature maps in the four downsampling stages to capture the long-range spatial dependence of important texture regions in the image; S3.2.5: Deploy a residual network module based on partial convolution on all feature maps of the second feature extraction branch. The partial convolution only performs spatial convolution on 1 / 4 of the input channels, and the remaining channels are identically mapped.
7. The method for identifying the sex of fresh cocoons and the inner dead cocoons by near-infrared multimodal feature fusion according to claim 6, characterized in that: Grouping normalization is applied after each convolutional layer of the first feature extraction branch; Batch normalization is applied after each convolutional layer in the second feature extraction branch; Both the first feature extraction branch and the second feature extraction branch use the same scaled Tanh activation function.
8. The method for identifying the sex of fresh cocoons and the inner dead cocoons by near-infrared multimodal feature fusion according to claim 1, characterized in that: The multimodal adaptive weighted fusion strategy includes at least the following steps: Adaptive average pooling is performed on the feature maps output from the first feature extraction branch and the feature maps output from the second feature extraction branch, respectively, and the pooled features are flattened into the first feature vector F. 1d With the second eigenvector F 2d ; Assign corresponding first learnable weights to the first and second feature vectors. With the second learnable weight The first and second learnable weights are normalized using the Softmax function. in, This is a natural exponential function, used to amplify the differences in the original weights; The learnable weights are the normalized weights of the first feature vector; The learnable weights are the normalized weights of the second feature vector; Multiply the normalized first learnable weight by the batch-normalized first feature vector to obtain the weighted first feature; The normalized second learnable weight is multiplied by the batch-normalized second feature vector to obtain the weighted second feature; The weighted first feature and the weighted second feature are concatenated to obtain the fused feature vector; in, This represents a vector concatenation operation; ; This represents element-wise multiplication; Indicates to Batch normalization operation; Indicates to Batch normalization operation.
9. The method for identifying the sex of fresh cocoons and the inner dead cocoons by near-infrared multimodal feature fusion according to claim 8, characterized in that: The classifier comprises a first fully connected layer, a Dropout layer, and a second fully connected layer connected in sequence. The input to the classifier is ; The first fully connected layer is used to reduce the dimension of the input fused feature vector to half; The dropout rate of the Dropout layer is set to 50%; The output of the second fully connected layer corresponds to the scores of three categories: female cocoons, male cocoons, and cocoons with internal dead printing. The classifier converts the score into class probabilities using the Softmax function; in, c is the natural exponential function; i This represents the output value of the i-th neuron in the final fully connected layer. It represents the total number of categories; It is the second fully connected layer of the classifier. The original output of each neuron, =1, 2, 3 correspond to female cocoons, male cocoons, and cocoons with inner dead printing, respectively.