A class imbalance image recognition method and system for wheat imperfect kernel recognition

Abstract

The application discloses a kind of class imbalance image recognition methods and systems for wheat imperfect kernel identification, build the recognition model of heterogeneous feature fusion decoupling, on the basis of existing residual network, including shared layer, expert layer, feature refining module, heterogeneous feature fusion module, imbalance optimization module and decoupling classification module, shallow layer feature output by shared layer is input into feature refining module after down-sampling and attention mechanism, and the expert layer feature output by expert layer is fused in heterogeneous feature fusion module, and the output result is linear layer and average is obtained;Mean input imbalance optimization module obtains first prediction result;The feature output by shared layer in the last stage is input into decoupling classification module, and second prediction result is obtained, and the first and second prediction results are weighted and summed as the final output of model.Compared with existing network model, the application has higher detection accuracy, and can still maintain stable recognition performance in real class imbalance data environment.

Description

Technical Field

[0001] This application belongs to the field of computer vision and grain quality detection technology, specifically relating to an image recognition method based on deep learning, and more particularly to a class imbalance image recognition method, system, electronic device and computer storage medium for automatic identification of imperfect wheat grains. Background Technology

[0002] Wheat is one of the world's most important food crops, and its yield and quality are closely related to national food security and public health. With the continuous advancement of modern agriculture, the demand for high-quality wheat suitable for industrial processing is steadily increasing. However, during harvesting, storage, and transportation, various imperfect grains, such as wilted, sprouted, broken, or moldy grains, not only deteriorate the processing performance of flour and reduce yield, but also pose a potential risk to food safety.

[0003] Early imperfect grain identification relied primarily on manual selection and visual inspection, resulting in low efficiency, high subjectivity, and poor repeatability, making it unsuitable for large-scale processing scenarios. With the development of image processing technology, researchers manually extracted features such as grayscale, color, and shape, combining them with classic methods like thresholding and edge detection for classification. However, these methods are highly sensitive to changes in lighting and data diversity, leading to insufficient robustness. The introduction of traditional machine learning algorithms has initially achieved automated identification, but limitations in the quality and universality of the designed features make it difficult to accurately identify complex and diverse types of imperfect grains, and their accuracy and generalization capabilities cannot meet practical needs. In recent years, deep learning, especially convolutional neural networks, has provided strong support for high-precision automatic identification of imperfect grains. Techniques such as data augmentation, transfer learning, model compression, and edge unrolling have further improved its practicality and deployment efficiency. Models such as MSPCNeXt, TLFF, ACE, Tade, BalPoE, and H2T have all demonstrated excellent performance in terms of identification accuracy and class balance.

[0004] However, in actual detection scenarios, the number of normal grains far exceeds that of imperfect grains. This data class imbalance leads to a decline in the recognition performance of the minority class (imperfect grains), thus affecting the robustness and practicality of the entire recognition system. Furthermore, existing models do not adequately utilize shallow features, failing to enrich the feature information of the tail category (imperfect grains). Due to their small sample size, subtle appearance features, and ease of confusion with normal grains, relying solely on deep abstract features is insufficient to construct a complete and robust category representation. Shallow features, on the other hand, can precisely reflect the subtle morphological differences of imperfect grains in terms of damage, lesions, and wrinkling. Existing models fail to effectively utilize this information to supplement and enhance the feature space of the tail category, resulting in insufficient feature expression and weak discriminative ability, ultimately leading to poor model recognition performance for the tail category. Summary of the Invention

[0005] To address the problems of insufficient accuracy in identifying imperfect wheat grains, inadequate utilization of shallow features, and class imbalance in existing technologies, this invention provides a class imbalance image recognition method and system for identifying imperfect wheat grains.

[0006] This invention is achieved through the following technical solution:

[0007] The first aspect of this invention provides a class imbalance image recognition method for identifying imperfect wheat grains, comprising:

[0008] Step S1: Collect wheat image data with labeled categories and divide it into training set and validation set;

[0009] Step S2: Build a heterogeneous feature fusion and decoupled recognition model, including a shared layer, an expert layer, a feature refinement module, a heterogeneous feature fusion module, an imbalanced optimization module, and a decoupled classification module;

[0010] Step S3: Input the data into the shared layer, output shallow features of different dimensions as input to the feature refinement module, perform downsampling through stride convolution, and then capture important information of the shallow features through the convolutional attention module;

[0011] Step S4: Input the output of the last stage of the shared layer into the expert layer to obtain expert layer features with the same number of shallow features. After being fused together with the shallow features output by the feature refinement module, the fusion is performed by the heterogeneous feature fusion module. After classification by the linear layer, the average of the results is taken as the output of the heterogeneous feature fusion module.

[0012] Step S5: Input the output of the heterogeneous feature fusion module into the imbalanced optimization module to obtain the prediction confidence of each category, construct the loss function, train the shared layer, expert layer, feature refinement module and heterogeneous feature fusion module with the goal of minimizing the loss function, and fix the optimized parameters.

[0013] Step S6: Create a new expert in the decoupled classification module, input the output of the last stage of the shared layer into the new expert to obtain expert features, and obtain the prediction confidence of each category through the classifier; based on the loss function of the imbalanced optimization module, introduce regularization to construct a total loss function, train the decoupled classification module with the goal of minimizing the total loss, and obtain the optimized parameters; the prediction results of the imbalanced optimization module and the decoupled classification module are weighted and summed as the final output of the model.

[0014] Step S7: The image to be predicted is input into the heterogeneous feature fusion decoupled recognition model. The shallow features output from the shared layer are input into the feature refinement module. The features output from the last stage of the shared layer are used to extract expert layer features by the expert layer and are also input into the decoupled classification module. The expert layer features and the output features from the feature refinement module are input into the heterogeneous feature fusion module. The result is input into the imbalance optimization module to obtain the first prediction result. If the first prediction result is a normal class, the result is used as the final output result of the model. If the first prediction result is an abnormal class, the result is weighted and summed with the second prediction result obtained from the decoupled classification module to obtain the final output result of the model.

[0015] In one specific embodiment, in step S1, data augmentation processing is performed on the collected data to increase the number of a small number of samples.

[0016] In one specific embodiment, in step S2, the heterogeneous feature fusion decoupling recognition model is improved based on the existing residual network. The convolutional and pooling layers and residual blocks (excluding the last layer) in the residual network are divided into shared layers. The last layer residual blocks are set as multiple identical residual blocks, i.e., expert parallelism, and defined as the expert layer. The shallow features output from the shared layer are input into the feature refinement module, and after downsampling and attention mechanisms, they are input together with the expert layer features output from the expert layer into the heterogeneous feature fusion module. After feature fusion, linear layers, and global average pooling and fully connected layers of the residual network, the results are output and averaged. The average result is input into the imbalance optimization module and processed by the residual network. The Softmax classifier is used to obtain the predicted classification and confidence score. The shared layer, expert layer, feature refinement module, and heterogeneous feature fusion module are trained and their optimized parameters are fixed using the constructed loss function. The decoupled classification module adds an expert, and the features output from the last stage of the shared layer are input into this expert. The expert layer features are then processed by the classifier to obtain the predicted class and probability. Regularization is introduced on the basis of the loss function of the imbalanced optimization module to suppress information from normal class data. The constructed total loss function is used to optimize the parameters of the decoupled classification module. The prediction results of the imbalanced optimization module and the decoupled classification module are weighted and summed to obtain the final output of the model.

[0017] In one specific embodiment, in step S3, the number of shallow features output by the shared layer is the same as the number of its residual blocks.

[0018] In one specific embodiment, in step S3, the convolutional attention module is composed of a channel attention module and a spatial attention module cascaded in series.

[0019] In one specific embodiment, in step S4, the expert layer includes multiple parallel residual blocks, and the number of residual blocks is the same as the number of residual blocks in the shared layer.

[0020] In one specific embodiment, the loss function in step S5 is as follows:

[0021]

[0022] in, Represents the total number of data within a single batch; The representative indicator function is the true label of data i. The value is 1 if it equals category c, and 0 otherwise. C represents the total number of categories in the dataset. Represents the category balance coefficient; Represents the focusing parameter; This represents the probability that the i-th data point is predicted to be of class c in the heterogeneous feature fusion module.

[0023] In one specific embodiment, in step S6, the total loss function is:

[0024]

[0025]

[0026]

[0027] in, This represents the weighting parameter of the regularization term. This represents the probability that the i-th data point is predicted to be in class c in the decoupled classification module. || represents the L2 norm, This represents the classification result of the i-th data in the decoupled classification module.

[0028] In one specific embodiment, in step S6, the weighted summation formula for the prediction results of the imbalance optimization module and the decoupled classification module is as follows:

[0029]

[0030] Where c=1 represents the normal class, Represents imperfect particles. These are the weighting coefficients.

[0031] A second aspect of the present invention provides a class imbalance image recognition system for identifying imperfect wheat grains, comprising:

[0032] The data acquisition module is used to collect wheat image data with labeled categories, dividing it into training and validation sets; it is also used to collect images to be predicted.

[0033] The shared layer is used to obtain shallow features of different dimensions by taking the output of the data acquisition module as input.

[0034] The expert layer is used to obtain the same number of expert layer features as the shallow layer features by using the output of the last stage of the shared layer.

[0035] The feature refinement module takes the shallow features output from the shared layer as input, performs downsampling through stride convolution, and then captures important information from the shallow features through the convolutional attention module.

[0036] The heterogeneous feature fusion module is used to fuse the shallow features output by the feature refinement module with the expert layer features, and then classify them through a linear layer, taking the average of the results.

[0037] The imbalanced optimization module is used to obtain the prediction confidence of each category by taking the output of the heterogeneous feature fusion module as input, constructing a loss function, and training the shared layer, expert layer, feature refinement module and heterogeneous feature fusion module with the goal of minimizing the loss function, and fixing the optimized parameters.

[0038] The decoupled classification module is used to create a new expert. It takes the output of the last stage of the shared layer as input to obtain expert features and obtains the prediction confidence of each category through a classifier. The total loss function is constructed and the decoupled classification module is trained with the goal of minimizing the total loss to obtain optimized parameters. The prediction results of the imbalanced optimization module and the decoupled classification module are weighted and summed as the final output of the model.

[0039] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the above-described method for class imbalance image recognition for identifying imperfect wheat grains.

[0040] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described class imbalance image recognition method for identifying imperfect wheat grains.

[0041] Compared with the prior art, this application has at least the following beneficial effects:

[0042] By introducing a decoupled classification mechanism, the constructed heterogeneous feature fusion decoupled recognition model focuses on learning imperfect granular features, effectively improving the accuracy of minority class recognition. Utilizing an imbalanced optimization strategy based on Focal Loss, the model pays more attention to difficult-to-classify data during training, mitigating performance bias caused by class imbalance. Through a feature refinement module and attention mechanism, the model's ability to capture information in key regions is enhanced, improving the targeting of feature recognition. A heterogeneous feature fusion mechanism effectively integrates feature information from different levels, improving the model's discriminative ability. By utilizing shallow features from the model's shared layers, the expression of class features is enriched, further optimizing the recognition effect.

[0043] The heterogeneous feature fusion and decoupling recognition model constructed in this invention has higher detection accuracy than existing network models, and it has a better improvement in the utilization of shallow network features and the recall rate of imperfect particles. It can still maintain stable recognition performance in real class imbalance data environment, and is suitable for actual grain detection scenarios. Attached Figure Description

[0044] Figure 1 : This is a flowchart of the prediction process for an example.

[0045] Figure 2 : This is a structural diagram of the network in the embodiment.

[0046] Figure 3 : This is a network structure diagram of the bottleneck of the residual block in the residual network 50 of the embodiment.

[0047] Figure 4 Comparison of class recall rates of InceptionNeXt, SwinV2, MSPCNeXt, H2T, and HIDIM on three datasets, where (a) is a comparison of class recall rates on the M600 dataset, (b) is a comparison of class recall rates on the P600 dataset, and (c) is a comparison of class recall rates on the G600 dataset. Detailed Implementation

[0048] To further understand the implementation method of this application, the technical solutions in the embodiments of this application are clearly and completely described. Furthermore, the examples described below are only a part of this application, not all of it. The following description is only for further illustrating the advantages and features of this application, and not for claiming limitations on the rights of this application. Other embodiments obtained by those skilled in the art without inventive effort are all within the scope of protection of this application.

[0049] A class imbalance image recognition method for identifying imperfect wheat grains, such as... Figure 1 As shown, it includes the following steps:

[0050] Step S1: Data Acquisition and Preprocessing.

[0051] We collected wheat image data with pre-labeled categories and divided it into training and validation sets. We then performed data augmentation on the collected data, including random brightness adjustment and horizontal flipping, to increase the number of samples and enrich sample diversity.

[0052] Step S2: Build a heterogeneous feature fusion and decoupling recognition model (HIDIM model).

[0053] The HIDIM model described is an improvement on the existing ResNet50 model, including a shared layer, an expert layer, a feature refinement module, a heterogeneous feature fusion module, an imbalanced optimization module, and a decoupled classification module, such as... Figure 2 As shown; the ResNet50 includes convolutional and max-pooling layers, four residual blocks, global average pooling and fully connected layers, and a Softmax classifier. The convolutional and pooling layers, as well as the first three residual blocks, are divided into shared layers. The last residual block is set as multiple identical residual blocks, i.e., expert parallelism, and defined as the expert layer. The shallow features output from the shared layer are input into the feature refinement module, undergo downsampling and attention mechanisms, and then input together with the expert layer features output from the expert layer into the heterogeneous feature fusion module. After feature fusion, linear layers, global average pooling, and fully connected layers, the results are output and averaged. The result mean is input into the imbalance optimization module and processed by So... The FTmax classifier obtains the predicted classification and confidence level. The shared layer, expert layer, feature refinement module, and heterogeneous feature fusion module are trained using a constructed loss function, and their optimized parameters are fixed. The decoupled classification module adds a new expert; the features output from the last residual block of the shared layer are input to this expert. The resulting expert layer features are then processed by the classifier to obtain the predicted class and probability. Regularization is introduced based on the loss function of the imbalanced optimization module to suppress information from normal class data. The constructed total loss function is used to optimize the parameters of the decoupled classification module. The prediction results from the imbalanced optimization module and the decoupled classification module are weighted and summed as the final output of the model.

[0054] Step S3: Data is input into Stage 1 (Conv1) of the shared layer, which is a convolutional and pooling layer with a size of H×W. This layer contains 64 convolutional kernels of size 7×7 with a stride of 2, and a 2×2 max pooling operation. The output size of Stage 1 is H / 4×W / 4. The H / 4×W / 4 data output from Stage 1 is input into Stage 2 (Conv2_x), which is the first residual block. This residual block contains 3 bottlenecks (e.g., ...). Figure 3 Each residual block bottleneck contains three convolutional layers with kernel sizes of 1×1, 3×3, and 1×1, respectively. The output size of stage 2 is H / 4×W / 4. The H / 4×W / 4 data output from stage 2 is input into stage 3Conv3_x, which is the second residual block. It contains four sequentially connected residual block bottlenecks identical to those in stage 1. The output size of stage 3 is H / 8×W / 8. The H / 8×W / 8 data output from stage 3 is input into stage 4Conv4_x, which is the third residual block. It contains six sequentially connected residual block bottlenecks identical to those in stage 1. The output size of stage 4 is H / 16×W / 16.

[0055] Shallow features output from stages 2 to 4 (Right now Figure 2The intermediate features in the model are used as input to the feature refinement module, and the formula is as follows:

[0056]

[0057] in, This represents the s-th stage of the shared layer, where s = 1, 2, 3, 4...S, and S represents the total number of stages. This represents the i-th input data. represent Enter the shared layer in the order of each stage. This represents the shallow feature of the s-th layer of the i-th data obtained.

[0058] The feature refinement module includes strided convolution and convolutional attention modules; the strided convolution consists of 3*3 convolutional layers with a stride of 2, providing shallow features. The stride convolution of the input feature refinement module achieves downsampling, downsampling the shallow feature dimensions to match the expert layer feature dimensions, as shown in the following formula:

[0059]

[0060] in represents the stride convolution of the s-th layer. Different stride convolutions of different numbers of layers are used for downsampling shallow features of different sizes. This represents the shallow features of the i-th data after downsampling.

[0061] Will get The input convolutional block attention module (CBAM) uses an attention mechanism to capture important information from shallow features. The CBAM module consists of two cascaded sub-modules: the Channel Attention Module (CAM) and the Spatial Attention Module (SAM).

[0062]

[0063] in, This represents the shallow feature output by the CBAM module for the i-th data point, with dimensions H / 32×W / 32.

[0064] Step S4: The expert layer comprises M parallel residual blocks, i.e., experts. The number of experts is the same as the number of stages in which the shallow features of the shared layer are output, i.e., M = S - 1. Each expert has the same residual block structure as the fourth layer of ResNet50, including 3 residual block bottlenecks. The features output by Conv4_x in the last stage of the shared layer are input into each expert in the expert layer. The input feature size is H / 16 × W / 16, and the feature size output by each expert in the expert layer is H / 32 × W / 32.

[0065] The obtained expert layer features [ , , ] and shallow features output by the feature refinement module [ , , After the heterogeneous feature fusion module performs corresponding fusion, the data is then classified through a linear layer:

[0066]

[0067] in, This represents the classification operation corresponding to the m-th expert layer feature; The expert layer feature represents the output of the i-th data from the m-th expert; This represents the output of the m-th expert layer for the i-th data. This represents the Hadamard Product; the mean of each result is calculated after global average pooling and fully connected layer processing, and is used as the output of the heterogeneous feature fusion module.

[0068] Step S5: Input the output of the heterogeneous feature fusion module into the imbalance optimization module. First, obtain the prediction confidence of each category through Softmax, which is used as the first prediction result.

[0069]

[0070] in, This represents the original result of the i-th data point and the c-th class, where C represents the total number of classes in the dataset; This represents the probability that the i-th data point is predicted to be of class c, and its value ranges from [0,1]; exp( The exponential function is used to convert numerical values ​​to positive numbers and amplify high-confidence outputs.

[0071] Construct and compute the Focal Loss function:

[0072]

[0073] in, Represents the total number of data within a single batch; The representative indicator function is the true label of data i. The value is 1 if it equals category c, and 0 otherwise. The class balance coefficient is used to assign weights to different classes to mitigate training bias caused by imbalanced data distribution. The Focal Loss parameter is used to reduce the loss weight of simple samples, so that the model can focus on learning difficult-to-classify samples and improve generalization ability. The parameters of the shared layer, expert layer, feature refinement module and heterogeneous feature fusion module are optimized with the goal of minimizing Focal Loss. The optimized parameters remain unchanged.

[0074] Step S6: Create a new expert in the decoupled classification module for training on the identification of imperfect granules. The new expert has the same structure as the experts in the expert layer. Input the features output from the last stage of the shared layer into the new expert. The obtained expert features are then classified by the classifier to obtain the classification result. Then, the prediction confidence of each category is obtained through Softmax. This serves as the second prediction result.

[0075] Based on the loss function of the imbalanced optimization module, regularization is introduced to suppress the interference of normal classes on the decoupled classification module, thus constructing the total loss function:

[0076]

[0077]

[0078]

[0079] in, The representative will Substituting the result into formula (6), The weighting parameter representing the regularization term is set to 0.001 in this embodiment; || || represents the L2 norm, used to avoid constraining features to be too large or too small, and to prevent gradient explosion; when c=1, it represents the normal category, so the decoupled classification module only considers imperfect granular data.

[0080] The decoupled classification module is trained with the goal of minimizing the total loss to obtain optimized parameters.

[0081] The first prediction result output by the imbalanced optimization module and the second prediction result output by the decoupled classification module are weighted and summed to obtain the final classification prediction result of the heterogeneous feature fusion decoupled recognition model:

[0082]

[0083] Where c=1 represents the normal class, and the final output of this class is the output of the imbalance optimization module; The imperfect particle represents the final prediction result, which is the weighted sum of the prediction results from the imbalanced optimization module and the decoupled classification module. These are the weighting coefficients.

[0084] Step S7: The image to be predicted is input into the heterogeneous feature fusion decoupled recognition model. After passing through the shared layer, three shallow features are output and input into the feature refinement module. The features output from the final stage of the shared layer are used to extract expert layer features and are also input into the decoupled classification module. The expert layer features and the output features from the feature refinement module are jointly input into the heterogeneous feature fusion module. The resulting product is input into the imbalance optimization module to obtain the first prediction result. If the first prediction result is a normal class, this result is used as the final output of the model. If the first prediction result is an abnormal class, it is weighted and summed with the second prediction result obtained from the decoupled classification module to obtain the final output of the model. The process is as follows: Figure 1 As shown.

[0085] Effect test:

[0086] Existing commonly used models were selected as comparison methods and tested against the model in this embodiment (HIDIM). Top-1 accuracy and Macro-F1 score were used as evaluation metrics to evaluate the model's prediction accuracy and minority class recognition performance, respectively. The comparison methods included SwinV2, InceptionNeXt, MSPCNeXt, and H2T. All models were tested on the same wheat dataset and trained using the Adam optimization algorithm. The computing resource used for training was a single NVIDIA GTX4090 GPU. All models were trained and tested on a single GTX4090 GPU.

[0087] The wheat datasets used specifically include the GrainSpace_Wheat_P600 dataset, the GrainSpace_Wheat_M600 dataset, and the GrainSpace_Wheat_G600 dataset. These three datasets are constructed based on images acquired by cameras with different performance levels, resulting in significant differences in image quality. The P600 is a professional-grade dual industrial camera, featuring fully automatic acquisition, high cost, and support for both vertical and horizontal perspective shooting; the G600 is a general-purpose single industrial camera, also fully automatic, with moderate cost, and single-view shooting; the M600 is a mobile smartphone with a fixed bracket, featuring purely manual acquisition, extremely low cost, single-view shooting, and no standardized light source. The GrainSpace_Wheat_M600 dataset includes a training set of 1,097,760 images, a validation set of 13,720 images, and a test set of 13,720 images. The GrainSpace_Wheat_P600 dataset includes a training set of 189,120 images, a validation set of 23,640 images, and a test set of 23,640 images. The GrainSpace_Wheat_G600 dataset includes a training set of 315,120 images, a validation set of 39,390 images, and a test set of 39,390 images. These datasets can be obtained from the GrainSpace website.

[0088] As shown in Table 1, the tests demonstrated high Top-1 accuracy on the M600, P600, and G600 datasets. However, the MacroF1 scores fluctuated more significantly, particularly on the G600 dataset where the F1 score was significantly lower, reflecting class imbalance or difficult-to-classify samples in this dataset. Overall, HIDIM achieved the best Top-1 accuracy and MacroF1 scores on all three datasets, significantly outperforming other comparative methods. InceptionNeXt and SwinV2 showed stable overall performance, while ConvNeXtV2 and H2T demonstrated outstanding F1 performance on the G600 dataset. TIFF and SHIKE, on the other hand, showed relatively weaker overall performance.

[0089] Table 1. Comparison of the effects of this embodiment with existing methods.

[0090]

[0091] In the recall comparison experiment, for the M600 dataset, such as Figure 4As shown in (a), this embodiment consistently achieved the highest recall across almost all categories. While the differences between models were negligible for normal class identification, significant variations occurred in the identification of the six imperfect granular categories. Notably, the recall rates for insect pests, mold, and germination were relatively low across all models, indicating a smaller sample size for these three categories. However, even under such class imbalance, this embodiment demonstrated superior results compared to the comparative methods. Performance on the P600 dataset was similar to that on the M600 dataset, as shown... Figure 4 As shown in (b), the recall differences between models are negligible in normal class identification. For imperfect grain identification, although this embodiment only slightly outperforms SwinV2 in the damaged and moldy categories, it achieves higher recall rates in other categories. Furthermore, in categories with relatively low recall rates, such as insect pests, black embryos, and Fusarium, this embodiment significantly outperforms other methods. Specifically, in the insect pest category, the recall rate is more than 4 percentage points higher than the second-best model, and the recall rate advantages in the black embryo and Fusarium categories reach 3.2 and 4.1 percentage points respectively, demonstrating strong robustness to low-sample, difficult-to-identify defect categories. This result fully verifies the classification advantage of this embodiment in class imbalance scenarios, effectively capturing key features of different defect categories, significantly improving the identification accuracy of each category, especially difficult-to-distinguish samples, and providing reliable technical support for the accurate detection of imperfect grains in agricultural products. For the G600 dataset, such as... Figure 4 As shown in c, the results differ slightly from the other two datasets. Our application performs best in the normal and Fusarium categories, while H2T has the highest recall in most other categories, ranking second. However, in the normal class, which has the largest sample size, H2T performs significantly worse than the other models. Therefore, our application still outperforms H2T in overall Top-1 accuracy and Macro-F1. Overall, these results confirm that our embodiment effectively addresses the class imbalance problem and improves the identification of hard samples.

[0092] Ablation studies:

[0093] To verify the effectiveness of each component, ablation studies were conducted by removing modules individually, and the following comparative models were constructed respectively:

[0094] Compared to Model 1: Excluding the feature refinement module, the rest of the model structure is the same as in this embodiment. This means that stride convolution and CBAM are removed, and intermediate features are aligned with expert layer features only through linear transformation.

[0095] Compared to Model 2: The heterogeneous feature fusion module is excluded, thereby eliminating the fusion between intermediate and expert functions. Therefore, the feature refinement module is also removed. The rest of the model structure is the same as in this embodiment.

[0096] Comparison Model 3: The standard cross-entropy loss is used to replace the focal loss to eliminate the unbalanced optimization module. The rest of the model structure is the same as in this embodiment.

[0097] Comparison Model 4: The decoupling classifier is removed to exclude the decoupling classification module; the rest of the model structure is the same as in this embodiment.

[0098] The ablation study results are shown in Table 2. This application achieved the best Top-1 accuracy and Macro-F1 score across all three datasets. Module analysis shows that on M600 and P600, removing the heterogeneous feature fusion module reduced the Macro-F1 score by 3.02 and 3.04 points, respectively, while removing the feature refinement module reduced it by 2.57 and 2.86 points. The Top-1 score changed by a maximum of 0.48 points, indicating that these two modules primarily improve the model's classification balance and generalization ability across classes, especially small sample classes, rather than simply increasing the overall classification accuracy. They effectively mitigate the performance bias caused by class imbalance and play a crucial role in improving the model's class-balanced classification performance. On G600, the Macro-F1 gap widened. Removing the heterogeneous feature fusion module reduced the Macro-F1 score by 11.11 points, and removing the feature refinement module reduced it by 6.67 points, highlighting the importance of heterogeneous fusion and feature refinement for minority class recognition under class imbalance. The decoupled classification module produces stable and moderate gains. After removal, the Macro-F1 scores of M600, P600, and G600 decreased by 0.95, 0.91, and 4.23, respectively, indicating that the decoupled classification module steepens the decision boundary, meaning that the model's ability to distinguish between different class features is significantly enhanced, and it can more clearly define class boundaries, thereby effectively improving the overall classification accuracy. The imbalanced optimization module has little impact on Top-1, but it continuously improves Macro-F1; its removal reduced the G600 Macro-F1 by 1.33 percentage points, showing that the imbalanced optimization module can specifically alleviate the bias caused by uneven data distribution, effectively improving the model's generalization performance and classification fairness on imbalanced classes without significantly affecting the overall recognition accuracy. Overall, these modules are complementary. HI and FR are the main drivers of macro-level balance. The decoupled classification module enhances stability, and the imbalanced optimization module further enhances imbalanced recall, together providing the best overall performance.

[0099] Table 2 Ablation Study Results

[0100]

[0101] During the parameter experiment, for the focusing parameter used as the focus loss parameter... The best settings are concentrated in =Approximately 2. On the M600, Top-1 is... Peak values ​​are reached at 2, 4, and 5, while the Macro-F1 reaches peak values ​​at 2, 4, and 5. The peak values ​​are reached at points 1, 2, and 3. On the P600, Top-1 is... =1,2 is optimal, Macro-F1 in =1.2,4 is optimal. On the G600, both the Top-1 and Macro-F1 are... The maximum value is reached when the value is equal to 2. In summary, =2 provides the most reliable performance on both datasets and metrics. With As the value increases, the measurement becomes less stable, which is consistent with the anode effect, i.e., a larger value... Emphasizing hard samples while suppressing easy samples can impair convergence and produce fluctuating results. On the M600, Top-1 is... It reaches its maximum value at 0.2, while the Macro-F1... The peak value is reached when =0. On the P600, Top-1 is at... =0 is best, while Macro-F1 is in Ideally, the value should be 0.2. On the G600, these two metrics are... The peak value is reached when the value is 0. The overall trend is across all datasets. Increasing this will decrease both metrics. This suggests that assigning larger weights to the (M+1)th decoupled classifier, i.e., using smaller... This can improve performance and demonstrate the effectiveness of the decoupled classifier module in identifying unhealthy wheat grains in this embodiment.

[0102] The implementation method of this application has been described in detail above, but this application is not limited to the above implementation method. Through a series of improvements, this application enhances the accuracy of identifying imperfect wheat grains, and has important application value in fields such as warehousing management, quality inspection, processing and utilization, grain trade, and agricultural informatization.

[0103] This embodiment also provides a class imbalance image recognition system for identifying imperfect wheat grains, including:

[0104] The data acquisition module is used to collect wheat image data with labeled categories, dividing it into training and validation sets; it is also used to collect images to be predicted.

[0105] The shared layer is used to obtain shallow features of different dimensions by taking the output of the data acquisition module as input.

[0106] The expert layer is used to obtain the same number of expert layer features as the shallow layer features by using the output of the last stage of the shared layer.

[0107] The feature refinement module takes the shallow features output from the shared layer as input, performs downsampling through stride convolution, and then captures important information from the shallow features through the convolutional attention module.

[0108] The heterogeneous feature fusion module is used to fuse the shallow features output by the feature refinement module with the expert layer features, and then classify them through a linear layer, taking the average of the results.

[0109] The imbalanced optimization module is used to obtain the prediction confidence of each category by taking the output of the heterogeneous feature fusion module as input, constructing a loss function, and training the shared layer, expert layer, feature refinement module and heterogeneous feature fusion module with the goal of minimizing the loss function, and fixing the optimized parameters.

[0110] The decoupled classification module is used to create a new expert. It takes the output of the last stage of the shared layer as input to obtain expert features and obtains the prediction confidence of each category through a classifier. The total loss function is constructed and the decoupled classification module is trained with the goal of minimizing the total loss to obtain optimized parameters. The prediction results of the imbalanced optimization module and the decoupled classification module are weighted and summed as the final output of the model.

[0111] This embodiment also provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the class imbalance image recognition method for identifying imperfect wheat grains provided in the above embodiment. The electronic device provided in this application embodiment can implement the various implementation methods in the above method embodiments, as well as the corresponding beneficial effects.

[0112] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the class imbalance image recognition method for identifying imperfect wheat grains provided in this application, and achieves the same technical effect.

[0113] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0114] The embodiments described above merely illustrate the implementation methods of this application, and should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A method for class-imbalanced image recognition for identifying imperfect wheat grains, characterized in that, include: Step S1: Collect wheat image data with labeled categories and divide it into training set and validation set; Step S2: Build a heterogeneous feature fusion and decoupled recognition model, including a shared layer, an expert layer, a feature refinement module, a heterogeneous feature fusion module, an imbalanced optimization module, and a decoupled classification module; Step S3: Input the data into the shared layer, output shallow features of different dimensions as input to the feature refinement module, perform downsampling through stride convolution, and then capture important information of the shallow features through the convolutional attention module; Step S4: Input the output of the last stage of the shared layer into the expert layer to obtain expert layer features with the same number of shallow features. After being fused together with the shallow features output by the feature refinement module, the fusion is performed by the heterogeneous feature fusion module. After classification by the linear layer, the average of the results is taken as the output of the heterogeneous feature fusion module. Step S5: Input the output of the heterogeneous feature fusion module into the imbalanced optimization module to obtain the prediction confidence of each category, construct the loss function, train the shared layer, expert layer, feature refinement module and heterogeneous feature fusion module with the goal of minimizing the loss function, and fix the optimized parameters. Step S6: Create a new expert in the decoupled classification module, input the output of the last stage of the shared layer into the new expert to obtain expert features, and obtain the prediction confidence of each category through the classifier; based on the loss function of the imbalanced optimization module, introduce regularization to construct the total loss function, and train the decoupled classification module with the goal of minimizing the total loss to obtain the optimized parameters; The prediction results of the imbalanced optimization module and the decoupled classification module are weighted and summed to obtain the final output of the model. Step S7: The image to be predicted is input into the heterogeneous feature fusion decoupled recognition model. The shallow features output from the shared layer are input into the feature refinement module. The features output from the last stage of the shared layer are used to extract expert layer features by the expert layer and are also input into the decoupled classification module. The expert layer features and the output features from the feature refinement module are input into the heterogeneous feature fusion module. The result is input into the imbalance optimization module to obtain the first prediction result. If the first prediction result is a normal class, the result is used as the final output result of the model. If the first prediction result is an abnormal class, the result is weighted and summed with the second prediction result obtained from the decoupled classification module to obtain the final output result of the model.

2. The method according to claim 1, characterized in that, In step S1, data augmentation processing is performed on the collected data to increase the number of a small number of samples.

3. The method according to claim 1, characterized in that, In step S2, the heterogeneous feature fusion decoupling recognition model is improved based on the existing residual network. The convolutional and pooling layers and residual blocks (except the last layer) in the residual network are divided into shared layers. The last residual block is set as multiple identical residual blocks in parallel and defined as an expert layer. The shallow features output from the shared layer are input into the feature refinement module, and after downsampling and attention mechanisms, they are input together with the expert layer features output from the expert layer into the heterogeneous feature fusion module. After feature fusion, linear layers, and global average pooling and fully connected layers of the residual network, the results are output and averaged. The average result is input into the imbalance optimization module, which is then processed by the residual network. The classifier obtains the predicted classification and confidence level. The shared layer, expert layer, feature refinement module, and heterogeneous feature fusion module are trained using a constructed loss function, and their optimized parameters are fixed. A new expert is added to the decoupled classification module. The features output from the final stage of the shared layer are input to this expert. The resulting expert layer features are then processed by the classifier to obtain the predicted category and probability. Regularization is introduced based on the loss function of the imbalanced optimization module to suppress information from normal class data. The constructed total loss function is used to optimize the parameters of the decoupled classification module. The prediction results from the imbalanced optimization module and the decoupled classification module are weighted and summed as the final output of the model.

4. The method according to claim 1 or 3, characterized in that, In step S3, the number of shallow features output by the shared layer is the same as the number of its residual blocks.

5. The method according to claim 1, characterized in that, In step S3, the convolutional attention module is composed of a channel attention module and a spatial attention module cascaded in series.

6. The method according to claim 1 or 3, characterized in that, In step S4, the expert layer includes multiple parallel residual blocks, and the number of residual blocks is the same as the number of residual blocks in the shared layer.

7. The method according to claim 1 or 3, characterized in that, In step S5, the loss function is as follows: in, Represents the total number of data within a single batch; The representative indicator function is the true label of data i. The value is 1 if it equals category c, and 0 otherwise. C represents the total number of categories in the dataset. Represents the category balance coefficient; Represents the focusing parameter; This represents the probability that the i-th data point is predicted to be of class c in the heterogeneous feature fusion module.

8. The method according to claim 1 or 3, characterized in that, In step S6, the total loss function is: in, This represents the weighting parameter of the regularization term. Represents the total number of data within a single batch; The representative indicator function is the true label of data i. The value is 1 if it equals category c, and 0 otherwise. C represents the total number of categories in the dataset. Represents the category balance coefficient; Represents the focusing parameter. This represents the probability that the i-th data point is predicted to be in class c in the decoupled classification module. || represents the L2 norm, This represents the classification result of the i-th data in the decoupled classification module.

9. The method according to claim 1 or 3, characterized in that, In step S6, the weighted summation formula for the prediction results of the imbalanced optimization module and the decoupled classification module is as follows: Where c=1 represents the normal class, Represents imperfect particles. These are the weighting coefficients. This represents the probability that the i-th data point is predicted to be of class c in the heterogeneous feature fusion module. This represents the probability that the i-th data point is predicted to be of class c in the decoupled classification module.

10. A class imbalance image recognition system for identifying imperfect wheat grains, used to implement the method described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to collect wheat image data with labeled categories and divide it into training and validation sets; It is also used to collect images to be predicted; The shared layer is used to obtain shallow features of different dimensions by taking the output of the data acquisition module as input. The expert layer is used to obtain the same number of expert layer features as the shallow layer features by using the output of the last stage of the shared layer. The feature refinement module takes the shallow features output from the shared layer as input, performs downsampling through stride convolution, and then captures important information from the shallow features through the convolutional attention module. The heterogeneous feature fusion module is used to fuse the shallow features output by the feature refinement module with the expert layer features, and then classify them through a linear layer, taking the average of the results. The imbalanced optimization module is used to obtain the prediction confidence of each category by taking the output of the heterogeneous feature fusion module as input, constructing a loss function, and training the shared layer, expert layer, feature refinement module and heterogeneous feature fusion module with the goal of minimizing the loss function, and fixing the optimized parameters. The decoupled classification module is used to create a new expert. It takes the output of the last stage of the shared layer as input to obtain expert features and obtains the prediction confidence of each category through a classifier. The total loss function is constructed and the decoupled classification module is trained with the goal of minimizing the total loss to obtain optimized parameters. The prediction results of the imbalanced optimization module and the decoupled classification module are weighted and summed as the final output of the model.

Images