A cocoon image classification method based on middle-level feature learning

Abstract

The application discloses a cocoon image classification method based on middle-layer feature learning, and compared with the prior art, the fine-grained problem of cocoon image data is improved, and comprises the following steps: cocoon image acquisition and dataset establishment; constructing a cocoon image classification module based on middle-layer feature learning; training of a cocoon image classification model; and classification test of a cocoon image test set. The application obtains a middle-layer attention feature map, divides the feature map region with a 7*7 window size, suppresses feature information in a significant feature region, increases the attention of the network to the defect distinguishable region distribution of the cocoon, effectively improves the feature redundancy phenomenon of the significant feature region, extracts the region features for fusion, captures more rich detail information for the cocoon feature extraction classification network, and compared with a ResNet50 basic classification model, the application improves 0.785%, and can be applied to a vision module of a cocoon sorting system.

Description

Technical Field

[0001] This invention patent relates to the fields of digital image processing and image classification, specifically to a method for classifying silkworm cocoon images based on mid-level feature learning. Background Technology

[0002] Silk reeling production practices show that due to differences in the physical condition of different silkworms and the conditions during cocoon spinning, or the influence of processes such as cocoon collection, drying, and transportation, it is inevitable that some cocoons that are unsuitable for silk reeling will be produced. Therefore, it is necessary to classify and select cocoons according to different process requirements, removing those unsuitable cocoons mixed in with the raw cocoons to meet the requirements of silk reeling.

[0003] Currently, factories still use manual cocoon sorting technology. While this can guarantee a certain level of accuracy, it requires a significant investment of manpower and time. Professional cocoon sorting workers need extensive training before they can start working, and even experienced workers with many years of experience are not immune to sorting errors.

[0004] Traditional image classification methods are ineffective because the feature differences between different categories in the original data are not obvious. In other words, the silkworm cocoon classification and recognition task has large intra-class feature differences and small inter-class feature differences, which makes it difficult for the network to extract detailed features of the silkworm cocoons and affects the classification accuracy.

[0005] To address the issue of fine-grained data in silkworm cocoon image data, a method is needed to enhance the network's ability to capture detailed features and improve classification accuracy, which can then be applied to the vision module of a silkworm cocoon sorting system. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention collects seven types of silkworm cocoon images and labels them accordingly. After preprocessing the images, data augmentation is performed to establish a silkworm cocoon image dataset. Based on the fine-grained nature of the dataset, a silkworm cocoon image classification model based on mid-level feature learning is constructed. The training set of silkworm cocoon images is input into the network for training to obtain the trained model. Finally, the trained model is used to classify a test set of silkworm cocoon images to test its classification accuracy. The specific technical solution is as follows:

[0007] A silkworm cocoon image classification method based on mid-level feature learning includes the following steps:

[0008] S1. Silkworm cocoon image acquisition and dataset creation;

[0009] S2. Construct a silkworm cocoon image classification model based on mid-level feature learning, and improve it by using a region saliency information suppression and feature fusion module at the mid-level features of the deep learning model ResNet-50;

[0010] S3. Training of the silkworm cocoon image classification model;

[0011] S4. Classification test of silkworm cocoon image test set.

[0012] Furthermore, in S1, an industrial camera was used to acquire images of seven types of silkworm cocoons: rotten cocoons, yellow cocoons, double-cocoons, thin-skinned cocoons, imprinted cocoons, and mouth cocoons. The cocoon images were preprocessed by performing grayscale conversion, Gaussian filtering, Otsu threshold segmentation, and connected component extraction. The minimum bounding rectangle was found, and the main body of the cocoon was extracted from the original image as a square based on the maximum length and width as the side length. Finally, the image size was adjusted to 448*448 to establish a silkworm cocoon image dataset.

[0013] Furthermore, the region saliency information suppression and feature fusion module in S2 processes the data using the following steps:

[0014] S21. Given an input feature map X∈R C×H×W Where C, H, and W represent the number of channels, height, and width of the input feature map, respectively; by applying average pooling and max pooling along the channel directions and stacking them, convolution is used to adjust the number of channels, and finally, an activation function is used to transform the input feature map X into an attention map X. att ∈R 1×H×W The attention score and the location information of salient features in the feature map are obtained, and their expressions are as follows:

[0015] X att =Sigmoid(f(concat([avg(X),max(X)])))

[0016] Where: f(·) represents a convolution operation with a 7×7 kernel size, concat(·) represents channel concatenation, avg(·) and max(·) represent the average pooling and max pooling functions in the channel direction, respectively, and Sigmoid(·) represents the Sigmoid activation function, which is used to obtain the attention map; in the attention score, each value represents the saliency of the feature information at each pixel location.

[0017] S22. Set the window size to 7×7 and include the feature attention map X. att The system is divided into multiple regions and implemented using convolutional layers with a kernel size of 7×7; X att Generate includes A set of attention intensities in regions of equal size Its expression is as follows:

[0018] P = Sigmoid(conv7-7(X) att ))

[0019] Where conv7_7(·) represents a convolutional layer operation with a kernel size of 7×7, and Sigmoid(·) represents the Sigmoid activation function.

[0020] S23. Select the position P(m, n) with the highest attention intensity from the attention intensity set P, and map it to the corresponding position in the output feature map X as follows:

[0021] IA={X(x,y)|7m≤x≤7(m+1), 7n≤y≤7(n+1), x∈N, y∈N}

[0022] IA∈R C×7×7 The region feature map in the input feature map X corresponding to the position P(m,n) where the attention intensity is the greatest;

[0023] S24. After obtaining the saliency of the feature information at each pixel location, the attention map X is then processed. att After applying a threshold mask, the threshold mask feature map T is obtained, and its expression is as follows:

[0024]

[0025] Where: X att (i, j) represents the attention map X. att The pixel value at position (i, j) in the region attention map is used to obtain the threshold mask feature map T, which corresponds to the attention map X at the position with the largest value in the region attention map. att Feature value suppression is applied to regions within the network, with a suppression level of 50%, to suppress features in salient regions, so that the network tends to learn feature information from other more salient regions.

[0026] S25. Using the obtained threshold mask feature map T as a weight vector, multiply it channel-by-channel with the output feature map X to obtain a feature map for suppressing regional saliency information. The data is fed into the next stage of network learning. The salient region feature map IA is flattened into one dimension after average pooling and then concatted with the output of the last layer of ResNet-50 before being fed into the classifier. It is then identified and classified through a fully connected layer and a softmax function.

[0027] Furthermore, the size of the attention intensity set P in S22 is Each region corresponds to an attention intensity. The attention intensity of each region is proportional to the discriminative power of that region, effectively approximating the spatial distribution of the most distinguishable regions. Thus, each region has a specific attention intensity.

[0028] Furthermore, in S3, the training set and validation set, along with their corresponding class labels, are fed into the network for training, with an input size of 448×448; the feature map R in the Conv4_6 output of the ResNet-50 model... m ∈R 1024×28×28 Then, using the region saliency suppression and feature fusion module, the mid-level feature map R... m After conversion to an attention map, the resulting partition generates a 4×4 grid feature map. The features of the most attention-prominent region among the 16 region feature maps are halved to obtain the salient region feature map IA∈R. 1024×7×7 Feature fusion with the output of the last layer of the ResNet-50 model enhances the model's representational ability.

[0029] Furthermore, in the training process parameter settings, the batch size and number of iterations were set to 12 and 80, respectively; the Adam optimizer was used, the initial learning rate was set to 0.0001, the cosine annealing algorithm was used to adjust the learning rate, the momentum was 0.95, and the weight decay coefficient was 0.0005.

[0030] Furthermore, in S4, the silkworm cocoon image to be tested undergoes the same preprocessing operation as the training data, and the image size is adjusted to be the same as the network input size. It is then input into the trained silkworm cocoon image classification model to obtain a probability classification vector. The category with the highest probability is the category to which the silkworm cocoon image to be tested belongs.

[0031] Beneficial effects:

[0032] This invention employs a mid-level feature learning-based silkworm cocoon image classification method. The mid-level feature map is pooled along the channel direction to obtain an attention feature map. Features containing highly saliency regions are extracted and fused to further improve the model's representational ability. Furthermore, feature information suppression is applied to saliency regions, which helps the network learn distinguishable features from other more saliency regions in the next stage, increasing the distribution of distinguishable regions and capturing richer detailed information for classification. Attached Figure Description

[0033] Figure 1 A flowchart illustrating the overall process of a silkworm cocoon image classification method based on mid-level feature learning, as provided in this application embodiment;

[0034] Figure 2 This is a flowchart of the silkworm cocoon image preprocessing provided in the embodiments of this application;

[0035] Figure 3 This is a structural diagram of the regional saliency information suppression and feature fusion module provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0037] like Figure 1 As shown, the silkworm cocoon image classification method based on mid-level feature learning of the present invention includes the following steps:

[0038] S1. Silkworm Cocoon Image Acquisition and Dataset Establishment

[0039] Using an industrial camera, after setting parameters such as aperture and exposure time, images were acquired using a ring light source. A total of 26,581 images of seven types of silkworm cocoons were collected based on the sorting criteria of frontline workers: cocoons on the machine, rotten cocoons, yellow cocoons, double cocoons, thin-skinned cocoons, imprinted cocoons, and mouth cocoons. The images were then labeled with category tags.

[0040] The acquired raw images are preprocessed, such as... Figure 2 As shown, traditional image processing techniques are used to sequentially perform grayscale conversion, Gaussian filtering, Otsu thresholding, and find the minimum bounding rectangle on the silkworm cocoon image. To suit the network input, the main body of the silkworm cocoon is extracted from the original image as a square based on the maximum length and width as the side length. The preprocessing operation removes a lot of black background and noise from the original image, highlighting the main body of the silkworm cocoon, which is beneficial for the network to extract features better. Finally, the image size is uniformly set to 448×448.

[0041] Image enhancement operations involve randomly rotating, scaling, distorting the color gamut, and randomly adjusting the saturation, brightness, contrast, and sharpness of a preprocessed image to enhance the image data.

[0042] The dataset is randomly divided into training, validation, and test sets in a ratio of 8:1:1.

[0043] S2. Construct a silkworm cocoon image classification model based on mid-level feature learning.

[0044] The silkworm cocoon image classification method based on mid-layer feature learning uses a network that is an improvement on the basic deep learning model ResNet-50. ResNet-50 residual network is a deep neural network characterized by its residual block structure, which addresses the vanishing and exploding gradient problems common in deep neural networks by increasing cross-layer connections. Silkworm cocoons are diverse, with different categories exhibiting varying degrees of difference in color, texture, and morphology, demanding strong feature representation capabilities from the network model. ResNet-50, with its high accuracy and trainability, is widely used in image classification.

[0045] In this embodiment, the structure diagram of the region saliency information suppression and feature fusion module is as follows: Figure 3 As shown, ResNet-50 consists of five stages: Conv_1, Conv_2, Conv_3, Conv_4, and Conv_5. The output feature maps of Conv_3, Conv_4, and Conv_5 can be defined as low-level, mid-level, and high-level feature maps, respectively.

[0046] This module can be added after stage3, stage4, and stage5. Its working principle is as follows:

[0047] S21. Given an input feature map X∈R C×H×W Where C, H, and W represent the number of channels, height, and width of the input feature map, respectively. To obtain the location information of salient features in feature map X, we apply average pooling and max pooling along the channel directions and stack them, adjust the number of channels using convolution, and finally use an activation function to transform the input feature map X∈R. C×H×W Convert to attention map X att ∈R 1×H×W Result in attention score X att =Sigmoid(f(X)), where: f(·) represents a convolution operation with a 7×7 kernel, concat(·) represents channel concatenation, avg(·) and max(·) represent the average pooling and max pooling functions in the channel direction, respectively, and Sigmoid(·) represents the Sigmoid activation function, thus obtaining the attention map. In the attention score, each value represents the saliency of the feature information at each pixel location.

[0048] S22. To extract more subtle features, feature maps at different levels can be further learned, thereby further improving the expressive power of the features. Set the window size to 7×7, and then... att This is achieved by dividing the data into multiple regions and using convolutional layers with a kernel size of 7×7. After processing using this method, X... att Generate a containing A set of attention intensities in regions of equal size Its expression is P = Sigmoid(conv7_7(X) att )).

[0049] The size of set P is Each region corresponds to an attention intensity. The attention intensity of each region is proportional to the discriminative power of that region, effectively approximating the spatial distribution of the most distinguishable regions. Thus, each region has a specific attention intensity.

[0050] S23. From the set of region attention intensities P, select the position P(m, n) with the highest attention intensity, and map it to the corresponding position in the output feature map X as follows:

[0051] IA={X(x,y)|7m≤x≤7(m+1), 7n≤y≤7(n+1), x∈N, y∈N}

[0052] IA∈R C×7×7 The region feature map in the input feature map X corresponds to the position P(m,n) where the attention intensity is the highest.

[0053] S24. After obtaining the saliency of the feature information at each pixel location, the attention map X is then processed. att After applying the threshold mask, the threshold mask feature map T is obtained:

[0054]

[0055] Where: X att (i, j) represents the attention map X. att The pixel value at position (i, j) in the region attention map is obtained. A threshold mask feature map T is then obtained, which corresponds to the attention map X at the position with the largest value in the region attention map. att The eigenvalues ​​of the regions are suppressed to a degree of 50% to suppress the features of the salient regions.

[0056] S25. Using the obtained threshold mask feature map T as a weight vector, multiply it channel-by-channel with the output feature map X to obtain a feature map for suppressing regional saliency information. Feature map X after suppressing regional saliency information r Sending the data to the next stage of the network's learning process can encourage the network to learn feature information from other more significant regions.

[0057] The output of the Conv4_6 convolution is the mid-level feature map R. m ∈R 1024×28×28 The region size is 7×7, and the feature map is R. m A 4×4 grid containing 16 regions was generated. The most salient region was found, and its features were halved and used as the input to the next layer of the network. A 7×7 region feature map was generated by extracting the feature map of the salient region for fusion.

[0058] In this embodiment, the output of the last layer of the ResNet-50 feature extraction network is 2048-dimensional. The salient region feature map IA is flattened into one dimension after average pooling and then concatted with the output of the last layer of ResNet-50 before being fed into the classifier. The classifier is then identified and classified through a fully connected layer and a softmax function.

[0059] S3. Training of the silkworm cocoon image classification model:

[0060] The silkworm cocoon image classification model based on mid-level feature learning is trained using the dataset of silkworm cocoon images to be trained; the process includes the following steps:

[0061] S31. Feed the training set and validation set, along with their corresponding class labels, into the network model for training. In each epoch of training, calculate the classification loss value using the cross-entropy loss function, backpropagate the loss value using gradient descent, update the network parameters with a certain learning rate, and then proceed to the next epoch of training. Adjust the learning rate until the network converges, and training is complete.

[0062] S32. The mid-layer feature map R in the Conv4_6 output of the ResNet-50 model m ∈R 1024×28×28 Then, using the region saliency suppression and feature fusion module, the mid-level feature map R... m After conversion to an attention map, the resulting partition generates a 4×4 grid feature map. The features of the most attention-prominent region among the 16 region feature maps are halved to obtain the salient region feature map IA∈R. 1024×7×7 Feature fusion with the output of the last layer of the ResNet-50 model enhances the model's representational ability;

[0063] S4. Classification test of silkworm cocoon image test set

[0064] The silkworm cocoon images in the test set undergo the same preprocessing operations as the training data, and the image size is adjusted to be the same as the network input. They are then input into the trained silkworm cocoon image classification network model to obtain probability classification vectors. The category with the highest probability is the category to which the silkworm cocoon image belongs.

[0065] To verify the impact of applying the region saliency suppression and feature fusion modules to different feature layers on the model's recognition accuracy, experiments were designed to compare the effects on different feature layers. Under the same experimental conditions, the low-level feature map R was compared. l ∈R 512×56×56 Mid-layer feature map R m ∈R 1024×28×28 High-level feature map R h ∈R 2048×14×14 Experiments were conducted using this module. Feature map R l Feature map R m Feature map R h After partitioning, 2×2, 4×4, and 8×8 grids were generated, containing 4, 16, and 64 regions, respectively. After extracting salient information regions, a 7×7 region feature map was generated for each of the low-level, mid-level, and high-level feature maps. The experimental results are shown in Table 1.

[0066] Model Low-level features Mid-layer features High-level characteristics Accuracy / % 1(ResNet-50) × × × 94.960 2 √ × × 95.657 3 × √ × 95.745 4 × × √ 95.069

[0067] Table 1 Experimental results of models with different feature layers

[0068] As shown in Table 1, compared to Model 1, Models 2-4 using this module all achieved improved accuracy on the test set. For high-level feature outputs, Model 4 using this module showed significantly lower accuracy than Models 2 and 3. This is because while high-level features have richer semantic information, their lower resolution and coarser target location contribute to the decreased accuracy. Model 2's accuracy is slightly lower than Model 3 because its lower-level features have higher resolution and contain more location and detail information, but their semantic content is lower and there is more noise.

[0069] Experiments have shown that by introducing the region saliency information suppression and feature fusion module to the end of the Conv4_6 feature layer in the ResNet-50 feature extraction network, the network model of this invention increases the distribution of distinguishable regions, makes full use of the features of silkworm cocoon surface defects, and the feature fusion enhances the model's representation ability, improving it by 0.785% compared with ResNet-50 classification.

[0070] The addition of region saliency suppression and feature fusion modules to the model increases the distribution of distinguishable regions, fully utilizes the features of defects on the silkworm cocoon surface, and effectively improves the feature redundancy phenomenon in saliency regions. Furthermore, the feature fusion method enhances the model's representational ability, allowing it to more accurately focus on the features of defective regions.

[0071] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A silkworm cocoon image classification method based on mid-level feature learning, characterized in that... Includes the following steps: S1. Silkworm cocoon image acquisition and dataset creation; S2. Construct a silkworm cocoon image classification model based on mid-level feature learning, and improve it by using region saliency information suppression and feature fusion modules at the mid-level features of the deep learning model ResNet-50; S3. Training of the silkworm cocoon image classification model; S4. Classification test of silkworm cocoon image test set; The region saliency information suppression and feature fusion module in S2 processes the data using the following steps: S21. Given the input feature map Where C, H, and W represent the number of channels, height, and width of the input feature map, respectively; by applying average pooling and max pooling along the channel directions and stacking them, convolution is used to adjust the number of channels, and finally, an activation function is used to transform the input feature map X into an attention map. The attention score and the location information of salient features in the feature map are obtained, and their expressions are as follows: in: This represents a convolution operation with a 7×7 kernel size. Indicates channel splicing. and These represent the average pooling and max pooling functions in the channel direction, respectively. express The activation function is used to obtain the attention map; in the attention score, each value represents the salience of the feature information at each pixel location. S22. Set the window size to 7×7 and add the feature attention map. It is divided into multiple regions and implemented using convolutional layers with a kernel size of 7×7; Generate includes P is a set of attention intensities in regions of equal size. Its expression is as follows: in, This indicates a convolutional layer operation with a kernel size of 7×7. express Activation function; S23. Select the position with the highest attention intensity from the set of attention intensities P. Mapped to the output feature map The corresponding position is: The location indicating the highest level of attention. Corresponding to the region feature map in the input feature map X; S24. After obtaining the saliency of the feature information at each pixel location, the attention map is then processed. After applying a threshold mask, the threshold mask feature map is obtained. Its expression is as follows: in: Attention map The position in the middle The pixel values ​​are used to obtain the threshold mask feature map. It will select the attention map corresponding to the position with the largest value in the region attention map. Feature value suppression is applied to regions within the network, with a suppression level of 50%, to suppress features in salient regions, so that the network tends to learn feature information from other more salient regions. S25. Obtain the threshold mask feature map As a weight vector, it is used in conjunction with the output feature map. Channel-by-channel multiplication yields a feature map with suppressed regional saliency. The feature maps of salient regions are fed into the next stage of network learning. After average pooling, the data is flattened into one dimension and then concatenated with the output of the last layer of ResNet-50 before being fed into the classifier. The data is then classified through a fully connected layer and a softmax function.

2. The silkworm cocoon image classification method based on mid-level feature learning as described in claim 1, characterized in that: S1 uses an industrial camera to acquire images of seven types of silkworm cocoons: rotten cocoons, yellow cocoons, double-cocoons, thin-skinned cocoons, imprinted cocoons, and mouth cocoons. The cocoon images are preprocessed by performing grayscale conversion, Gaussian filtering, Otsu threshold segmentation, and connected component extraction. The minimum bounding rectangle is found, and the main body of the cocoon is extracted from the original image as a square based on the maximum length and width as the side length. Finally, the image size is adjusted to 448*448, and a silkworm cocoon image dataset is established.

3. The silkworm cocoon image classification method based on mid-level feature learning as described in claim 1, characterized in that: Set of attention intensity in S22 Size is (1, , Each region corresponds to an attention intensity; the attention intensity of each region is proportional to the discrimination ability of that region, which can effectively approximate the spatial distribution of the most distinguishable regions; at this time, each region has an attention intensity.

4. The silkworm cocoon image classification method based on mid-level feature learning as described in claim 1, characterized in that: In S3, the training and validation sets, along with their corresponding class labels, are fed into the network for training. The input size is 448×448. The Conv4_6 output of the ResNet-50 model contains the mid-layer feature maps. Then, the region saliency information suppression and feature fusion module is used to process the mid-level feature map. After conversion to an attention map, the resulting 4×4 grid feature maps were generated. The features of the most attention-prominent region among the 16 region feature maps were halved to obtain the salient region feature map. Feature fusion with the output of the last layer of the ResNet-50 model enhances the model's representational capabilities.

5. The silkworm cocoon image classification method based on mid-level feature learning as described in claim 4, characterized in that: During training, the batch size and number of iterations were set to 12 and 80, respectively. The Adam optimizer was used, with an initial learning rate of 0.0001. The cosine annealing algorithm was used to adjust the learning rate, with a momentum of 0.95 and a weight decay coefficient of 0.0005.

6. The silkworm cocoon image classification method based on mid-level feature learning as described in claim 1, characterized in that: In S4, the silkworm cocoon image to be tested undergoes the same preprocessing operations as the training data, and the image size is adjusted to be the same as the network input size. It is then input into the trained silkworm cocoon image classification model to obtain a probability classification vector. The category with the highest probability is the category to which the silkworm cocoon image to be tested belongs.

Images