Visual counting method for knowledge distillation of kiwi fruits in 1+2 +1 distillation mode

Abstract

The invention belongs to the technical field of image recognition, and particularly relates to a knowledge distillation kiwi fruit visual counting method in a 1 + 2 + 1 distillation mode, and the method comprises the steps: building a student model through a YOLOv11 network model; taking the SEAM attention module as a detection head of the YOLOv11 network model to obtain a teacher model; an SBA self-adaptive attention module, an SOEP module and two teaching assisting models are added to a neck network of the YOLOv11 network model; distilling the two teaching assistant models by using the teacher model to obtain two distillation loss functions; and carrying out distillation training on the student model by using the sum of the two distillation loss functions, then carrying out self-distillation operation on the student model, and carrying out detection and counting on the input kiwi fruit images by using the student model after distillation training. According to the invention, the problem that the gap between a teacher model and a student model is too large can be solved, the reliability and accuracy of the model are improved, and efficient kiwi fruit counting is realized.

Description

Technical Field

[0001] The invention belongs to the technical field of image recognition and relates to a knowledge distillation kiwi fruit visual counting method in a 1+2+1 distillation mode. Background Art

[0002] Kiwifruit is a temperate and subtropical fruit that is rich in nutrients and has a unique flavor. Although it is popular among consumers and growers, its growth characteristics in the natural environment, such as overlapping fruits, tree branches blocking the view, and changing light, pose a huge challenge to yield statistics. Traditional manual counting methods are not only time-consuming and labor-intensive, but also costly and difficult to guarantee accuracy. Therefore, the development of convenient and efficient kiwifruit counting technology has become an urgent problem to be solved in digital agricultural production.

[0003] The rapid development of deep learning technology has provided new possibilities for solving this problem. A series of algorithms designed for fruit detection, identification and positioning have been introduced into the agricultural field. These methods provide strong support for precision agriculture with their reliable performance, strong generalization ability and excellent environmental adaptability. As the core means to achieve accurate counting of crops, computer vision technology can significantly reduce manpower input and improve the accuracy of yield statistics. At the same time, it can help growers optimize production decisions, increase yields and reduce production costs, thereby promoting the sustainable development of agricultural production.

[0004] Although existing research has made progress in the kiwifruit counting task, there are still some limitations that need to be improved. First, the dataset needs to be expanded to cover more diverse kiwifruit varieties and types. Second, the network still has room for improvement in dealing with occlusion, small objects, and multi-scale research. Finally, the knowledge distillation method used has shortcomings such as long distillation time and large amount of calculation. Summary of the invention

[0005] The purpose of the present invention is to overcome the limitations of the prior art and propose a knowledge distillation kiwifruit visual counting method based on a 1+2+1 distillation mode. The 1+2+1 distillation mode refers to a method in which a teacher model and two assistant teaching models perform knowledge distillation training on a student model. The student model is optimized through this mode. This method can reduce the workload of fruit farmers, improve the counting efficiency of kiwifruit, and save labor costs.

[0006] The present invention is implemented by the following scheme: A knowledge distillation kiwi fruit visual counting method in a 1+2+1 distillation mode, comprising: Build a student model using the YOLOv11 network model; The SEAM attention module is used as the detection head of the YOLOv11 network model to obtain the teacher model; An SBA adaptive attention module is added to the neck network of the YOLOv11 network model to splice features of different scales, thereby obtaining the teaching assistant model I; the SBA adaptive attention module includes two calibration attention units, and the predicted feature maps output by the two calibration attention units are spliced ​​and convolved to obtain the output of the SBA adaptive attention module; A SOEP module is added to the neck network of the YOLOv11 network model to obtain the teaching assistant model II, wherein the SOEP module includes a CSP-OmniKernel module and a spatial depth conversion convolution; The teacher model is used to distill the teaching assistant model I to obtain the distillation loss function I; the teacher model is used to distill the teaching assistant model II to obtain the distillation loss function II; The student model is distilled and trained using the sum of distillation loss function I and distillation loss function II to generate distillation loss function III. The student model is then self-distilled and the input kiwi images are detected and counted using the student model trained by self-distillation.

[0007] Further preferably, in the neck network of the teaching assistant model I, the features of each layer of the backbone network are firstly spliced ​​from the deep layer to the shallow layer through the SBA adaptive attention module, and the spliced ​​features are fused through the feature fusion module; then, the fused features are gradually spliced ​​from the shallow layer to the deep layer through the SBA adaptive attention module, and the spliced ​​features are fused through the feature fusion module, and the fused features of different levels are used as inputs of different detection heads.

[0008] Further preferably, the SBA adaptive attention module includes two calibration attention units, both input features are processed by the two calibration attention units, and the predicted feature maps output by the two calibration attention units are concatenated and then subjected to 3×3 convolution to obtain the output of the SBA adaptive attention module.

[0009] Further preferably, the SOEP module includes a CSP-OmniKernel module and a spatial depth conversion convolution (SPDConv). In the neck network of the teaching assistant model II, shallow features and deep features are sampled and then processed by the spatial depth conversion convolution. The features output by the spatial depth conversion convolution are then spliced ​​with the shallow features, and the spliced ​​features are then processed by the CSP-OmniKernel module.

[0010] Further optimization is to use the teacher model to train the collected kiwifruit images to generate the knowledge soft targets of the teacher model, and then generate the soft labels of the teacher model from the knowledge soft targets of the teacher model. Finally, the distillation loss function I is generated by the soft labels of the teacher model and the soft labels of the teaching assistant model I; the distillation loss function II is generated by the soft labels of the teacher model and the soft labels of the teaching assistant model II.

[0011] Further preferably, the student model after distillation training generates a knowledge soft target for the input kiwi fruit image, and then generates a soft label from the knowledge soft target, and calculates the student model loss, and finally obtains a hard target. The hard target is used to predict the kiwi fruit image to obtain a recognition box, and the recognition box is counted, which is the number of kiwi fruits.

[0012] Further preferably, the student model is trained using a kiwi fruit image dataset to generate a knowledge soft target, and then the soft label is generated from the knowledge soft target.

[0013] The present invention uses a teacher model, and the SEAM module is added to the head of the teacher model. As a teacher model, the student model can improve the recognition of occlusion of the target object. Then, in order to make up for the gap between the teacher model and the student model, the teaching assistant model is introduced. The teaching assistant model is to add the SBA module to the neck network of the YOLOv11 network model, as the teaching assistant model I, which can improve the student model's recognition of small targets; add the CSP-OmniKernel module to the neck network of the YOLOv11 network model, as the teaching assistant model II, to improve the module's recognition of small targets. Finally, the student model is self-distilled to improve the performance of the student model. This method is helpful for counting kiwifruit and has important theoretical significance and practical value in the field of agriculture. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the method of the present invention.

[0015] Figure 2 These are example diagrams of the kiwifruit image dataset of the present invention, where (a) is an image of a ripe fruit under strong light, (b) is an image of an unripe fruit under strong light, (c) is an image occluded by branches, (d) is an image occluded by leaves, (e) is an image of a small target, and (f) is a multi-scale image.

[0016] Figure 3 This is the YOLOv11 network model architecture diagram.

[0017] Figure 4 This is the architecture diagram of the teacher model YOLOv11-SEAM.

[0018] Figure 5 This is the architecture diagram of YOLOv11-SBA.

[0019] Figure 6 This is the architecture diagram of YOLOv11-SOEP.

[0020] Figure 7 Schematic diagram of the SBA adaptive attention module. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] like Figure 1 As shown, a knowledge distillation kiwi fruit visual counting method based on a 1+2+1 distillation mode includes the following steps 1 to 8.

[0023] Step 1: Collect kiwi images and build a kiwi image dataset; The kiwi images were manually labeled to ensure that the kiwis were all within the closed rectangular bounding box. Finally, 4336 kiwi images with a resolution of 640×640 pixels were selected (e.g. Figure 2 As shown in Figure 1, the total kiwifruit image dataset is randomly divided into training set, validation set and test set in a ratio of 7:2:1. Table 1 shows the detailed information of the division of the kiwifruit image dataset. Table 1

[0024] Step 2: Build a student model using the YOLOv11 network model, train the student model using the kiwi image dataset, generate knowledge soft targets, and then generate soft labels from the knowledge soft targets; Use the YOLOv11 network model, modify the model training parameters and yaml configuration file to make it a student model module; The structure of the YOLOv11 network model is as follows Figure 3As shown, it includes a backbone network, a neck network and a head network. The backbone network includes a first convolution module, a second convolution module, a first feature fusion module, a third convolution module, a second feature fusion module, a fourth convolution module, a third feature fusion module, a fifth convolution module, a fourth feature fusion module, a spatial pyramid pooling module, a channel and a spatial attention module connected in sequence; the neck network includes a first upsampling module, a first splicing module, a fifth feature fusion module, a second upsampling module, a second splicing module, a sixth feature fusion module, a sixth convolution module, a third splicing module, a seventh feature fusion module, a seventh convolution module, and an eighth feature fusion module. The head network includes a first detection head, a second detection head and a third detection head. After the features output by the channel and spatial attention module are processed by the first upsampling module, they are spliced ​​with the features output by the first splicing module and the third feature fusion module, and the spliced ​​features are fused by the fifth feature fusion module. The fused features output by the fifth feature fusion module are processed by the second upsampling module, and then spliced ​​with the features output by the second feature fusion module in the second splicing module. The spliced ​​features are fused by the sixth feature fusion module, and the fused features output by the sixth feature fusion module are used as the input of the first detection head. The fused features output by the sixth feature fusion module are also processed by the sixth convolution module, and the features output by the sixth convolution module are spliced ​​with the fused features output by the fifth feature fusion module in the third splicing module. The spliced ​​features are input to the seventh feature fusion module for feature fusion, and the fused features output by the seventh feature fusion module are used as the input of the second detection head. The fused features output by the seventh feature fusion module are also processed by the seventh convolution module, and the features output by the seventh convolution module are spliced ​​with the features output by the channel and spatial attention module in the fourth splicing module. The spliced ​​features are input to the eighth feature fusion module for feature fusion, and the fused features output by the eighth feature fusion module are used as the input of the third detection head.

[0025] The main processing flow of the YOLOv11 network model is as follows: First, the image is resized to 640×640×3 through image preprocessing, and then the image is input into the backbone network for feature extraction. The result of feature extraction is to generate features with sizes of 80×80×256, 40×40×512, and 20×20×1024. Subsequently, the three feature layers are fused in the neck network to form three new fused features. Finally, all fused features are sent to the three detection heads to complete the target detection task.

[0026] It should be noted that the numbering of the modules such as "first, second, third" in this application is only for the convenience of explaining the model architecture. The modules with different numbers have the same structure but different parameters. The structure of the YOLOv11 network model is a publicly available prior art and will not be further elaborated here.

[0027] Step 3: Use the SEAM attention module as the detection head of the YOLOv11 network model to obtain the teacher model YOLOv11-SEAM, as shown in Figure 4 As shown; Since the occlusion of obstacles can easily cause incomplete features and misjudgment of the number of fruits, in order to achieve multi-scale kiwi detection, enhance the saliency of kiwis in images, and suppress background interference, the SEAM attention module is introduced to enhance the robustness and generalization ability of the model through multi-view feature fusion and consistency regularization. The SEAM attention module starts with a residual connected depth separable convolution operation, and then divides it according to the number of channels. Although this method reduces the number of parameters and knows the importance of each channel, it ignores the information interaction between channels. To solve this problem, a point (1×1) convolution is combined with the output of separable convolutions of different depths. After that, a two-layer fully connected network is used to allow all channels to communicate with each other. The loss is compensated by learning the relationship between occluded kiwis and unoccluded kiwis. The exponential function is used to process the original importance score learned by the fully connected layer, mapping the value from [0, 1] to [1, e], where e is a natural constant. This mapping relationship improves the tolerance of position errors. Finally, the output of the SEAM attention module is multiplied with the attention of the original features to make the teacher model more effective for kiwi occlusion.

[0028] Step 4: Add the SBA adaptive attention module to the neck network of the YOLOv11 network model to splice features of different scales, and obtain the YOLOv11-SBA network model as the teaching assistant model I; The structure of YOLOv11-SBA is as follows Figure 5As shown in FIG. 1 , the backbone network is the same as the backbone network of the YOLOv11 network model, the head network has 4 detection heads, and in the neck network, the features output by the third feature fusion module, the spatial pyramid pooling module, the channel and spatial attention module are spliced ​​by the first SBA adaptive attention module, the features spliced ​​by the first SBA adaptive attention module are fused in the fifth feature fusion module, the fused features output by the fifth feature fusion module are spliced ​​with the features output by the second feature fusion module in the second SBA adaptive attention module, the features spliced ​​by the second SBA adaptive attention module are fused in the sixth feature fusion module, the fused features output by the sixth feature fusion module are spliced ​​with the features output by the first feature fusion module in the third SBA adaptive attention module, the features spliced ​​by the third SBA adaptive attention module are fused in the ninth feature fusion module, and the fused features output by the ninth feature fusion module are spliced ​​with the features spliced ​​by the second SBA adaptive attention module. The connected features are spliced ​​in the fourth SBA adaptive attention module, the features spliced ​​by the fourth SBA adaptive attention module are fused in the seventh feature fusion module, the fused features output by the seventh feature fusion module and the features spliced ​​by the first SBA adaptive attention module are spliced ​​in the fifth SBA adaptive attention module, the features spliced ​​by the fifth SBA adaptive attention module are fused in the tenth feature fusion module, the fused features output by the tenth feature fusion module and the features output by the spatial pyramid pooling module, the channel and spatial attention module are spliced ​​in the sixth SBA adaptive attention module, and the features spliced ​​by the sixth SBA adaptive attention module are fused in the eighth feature fusion module; the fused features output by the ninth feature fusion module, the fused features output by the seventh feature fusion module, the features spliced ​​by the fifth SBA adaptive attention module, and the fused features output by the eighth feature fusion module are selected as the input of the four detection heads of the head network.

[0029] Different from the previous fusion method, the SBA adaptive attention module includes two calibration attention units (RAUs). The predicted feature maps output by the two calibration attention units are concatenated and then 3×3 convolved to obtain the output of the SBA adaptive attention module.

[0030] The calibrated attention unit (RAU) mutually adapts the input features before fusion. Figure 7 As shown in Figure 2, two calibrated attention units (RAUs) input deep and shallow information in different ways to compensate for the lack of shallow semantic information and the lack of spatial boundary information in the deep layer. The calibrated attention unit combines different features and improves incomplete features. and There are two input features. Through two linear mappings and activation functions, the input features are processed, and the number of channels is reduced to 32 to obtain the feature map and , and then the uncertain and rough estimates are turned into a complete prediction map through dot products and inverse operations. The process of calibrating the attention unit is expressed as: ; ; in, The predicted feature map output by one of the calibrated attention units; are two input features, is the deep semantic information of 40×40×512 dimensions, is the shallow semantic information of 80×80×256 dimensions, for The feature map obtained after processing is for The feature map obtained after processing. and are two sigmoid activation functions, and ⊙ represents point-by-point multiplication.

[0031] The fusion process of the SBA adaptive attention module is expressed as: ; in represents a 3×3 convolution followed by a batch normalization and RELU activation layer. is the predicted feature map output by another calibration attention unit, Concat represents the concatenation operation. Z represents the feature map output by the SBA adaptive attention module, with a dimension of 80×80×256.

[0032] Step 5: Add the SOEP module to the neck network of the YOLOv11 network model to obtain the YOLOv11-SOEP network model as the teaching assistant model II; like Figure 6 As shown, the SOEP module includes a CSP-OmniKernel module and a spatial depth conversion convolution (SPDConv). In the neck network of the YOLOv11 network model, the features output by the second feature fusion module and the fifth feature fusion module are sampled by the second upsampling module and then processed by the spatial depth conversion convolution (SPDConv). The features output by the spatial depth conversion convolution are then spliced ​​with the features output by the second feature fusion module in the second splicing module. The spliced ​​features are then processed by the CSP-OmniKernel module, and the features output by the CSP-OmniKernel module are fused through the sixth feature fusion module.

[0033] Furthermore, in the target detection task using YOLOv11, it is generally difficult to process small targets on the normal detection layer. The traditional improvement is to add a second convolution module to improve the detection ability of small targets, but it also brings some problems, such as increased computational complexity and more time-consuming post-processing after adding the second convolution module. In order to solve the problem and improve efficiency, the neck network is improved, and the CSP-OmniKernel module is created based on the CSP module and the OmniKernel module, and the SOEP module is proposed. We did not directly add the second convolution module, but used the second feature fusion module to obtain features rich in small target information through spatial depth conversion convolution (SPDConv), and then fused it with the sixth feature fusion module.

[0034] Step 6: Use the teacher model to distill the teaching assistant model I. First, use the teacher model to train the collected kiwifruit images to generate the knowledge soft targets of the teacher model, then generate the soft labels of the teacher model from the knowledge soft targets of the teacher model, and finally generate the distillation loss function I through the soft labels of the teacher model and the soft labels of the teaching assistant model I; Step 7: Use the teacher model to distill the teaching assistant model II. First, use the teacher model to train the collected kiwifruit images to generate the knowledge soft targets of the teacher model, then generate the soft labels of the teacher model from the knowledge soft targets of the teacher model, and finally generate the distillation loss function II through the soft labels of the teacher model and the soft labels of the teaching assistant model II; Step 8: Perform self-distillation on the student model, and use the student model after distillation training to detect and count the input kiwifruit images. Use the sum of distillation loss function I and distillation loss function II to perform distillation training on the student model, generate distillation loss function III, and then perform self-distillation on the student model to generate distillation loss function IV; the student model after self-distillation training generates knowledge soft targets for the input kiwifruit images, and then generates soft labels from the knowledge soft targets, and calculates the student model loss, and finally obtains hard targets, uses the hard targets to predict the kiwifruit images, obtains the recognition boxes, and counts the recognition boxes, which is the number of kiwifruit.

[0035] In order to improve the accuracy of the model, more networks and parameters are usually introduced. The result is that the model becomes more complex, but complex models are not suitable for deployment in devices with limited computing. Knowledge distillation can train complex models by softening the output, thereby improving the accuracy of the model. However, in most cases, the learning ability of the student model is low and many changes cannot be observed. The teaching assistant model is introduced to solve this problem. The learning ability and scale of the teaching assistant model are between the teacher model and the student model, which can help the student model learn better.

[0036] This application uses YOLOv11-SEAM, which has a large number of parameters and precision, as the teacher model for pre-training; secondly, YOLOv11-SBA and YOLOv11-SOEP network models are used as teaching assistant model I and teaching assistant model II, respectively, and the teacher model is used to distill the teaching assistant model I to obtain the distillation loss function I; then the teacher model is used to distill the teaching assistant model II to obtain the distillation loss function II, and the sum of the distillation loss function I and the distillation loss function II is used to perform distillation training on the student model to generate the distillation loss function III, and then the student model is self-distilled to generate the distillation loss function IV, so that the student model can obtain more knowledge and improve the generalization ability of the student model. The training results are shown in Table 2. TSS means that the teacher model directly performs knowledge distillation on the student model, and then the student model performs self-distillation. T-TA1-SS means that before the teacher model performs knowledge distillation on the student model, it must first perform knowledge distillation on the teaching assistant model I, and then the teaching assistant model I performs knowledge distillation on the student model, and finally performs self-distillation. T-TA2-SS means that before the teacher model performs knowledge distillation on the student model, it must first perform knowledge distillation on the teaching assistant model II, and then the teaching assistant model II performs knowledge distillation on the student model, and finally performs self-distillation. The evaluation parameters P represent precision, R represent recall, mAP50 represent average precision, and mAP50:95 represent a series of IoU thresholds from 0.5 to 0.95.

[0037] Table 2

[0038] In order to further explore the performance advantages and disadvantages of the present invention and the current excellent target detection algorithms, this embodiment trained YOLOv11, YOLOv10, YOLOv9, and YOLOv8 under the same experimental environment. The results are shown in Table 3. The present invention shows better performance. Compared with other single-stage target detection algorithms, its average precision (mAP50) is 1.3%, 4.1%, 2.5%, and 1.9% higher than that of YOLOv11, YOLOv10, YOLOv9, and YOLOv8, respectively; its precision (Precision) is 1.1%, 2.9%, 2.2%, and 0.1% higher, respectively. These results highlight the advantages of the present invention.

[0039] Table 3

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge distillation kiwi fruit visual counting method based on a 1+2+1 distillation model, characterized in that: include: Build a student model using the YOLOv11 network model; The SEAM attention module is used as the detection head of the YOLOv11 network model to obtain the teacher model; An SBA adaptive attention module is added to the neck network of the YOLOv11 network model to splice features of different scales, thereby obtaining the teaching assistant model I; the SBA adaptive attention module includes two calibration attention units, and the predicted feature maps output by the two calibration attention units are spliced ​​and convolved to obtain the output of the SBA adaptive attention module; A SOEP module is added to the neck network of the YOLOv11 network model to obtain the teaching assistant model II, wherein the SOEP module includes a CSP-OmniKernel module and a spatial depth conversion convolution; The teacher model is used to distill the teaching assistant model I to obtain the distillation loss function I; the teacher model is used to distill the teaching assistant model II to obtain the distillation loss function II; The student model is distilled and trained using the sum of distillation loss function I and distillation loss function II to generate distillation loss function III. The student model is then self-distilled and the input kiwi images are detected and counted using the student model trained by self-distillation.

2. The knowledge distillation kiwifruit visual counting method according to claim 1, characterized in that: In the neck network of the teaching assistant model I, the features of each layer of the backbone network are first spliced ​​from deep to shallow through the SBA adaptive attention module, and the spliced ​​features are fused through the feature fusion module; then the fused features are spliced ​​from shallow to deep through the SBA adaptive attention module, and the spliced ​​features are fused through the feature fusion module. The fused features of different levels are used as inputs of different detection heads.

3. The knowledge distillation kiwifruit visual counting method according to claim 1, characterized in that: In the neck network of the assistant model II, shallow features and deep features are sampled and then processed by spatial depth conversion convolution. The features output by the spatial depth conversion convolution are then concatenated with the shallow features, and the concatenated features are processed by the CSP-OmniKernel module.

4. The knowledge distillation kiwifruit visual counting method according to claim 1, characterized in that: The collected kiwifruit images are trained using the teacher model to generate the knowledge soft targets of the teacher model, and then the soft labels of the teacher model are generated from the knowledge soft targets. Finally, the distillation loss function I is generated by the soft labels of the teacher model and the soft labels of the teaching assistant model I; the distillation loss function II is generated by the soft labels of the teacher model and the soft labels of the teaching assistant model II.

5. The knowledge distillation kiwifruit visual counting method according to claim 1, characterized in that: The student model after distillation training generates knowledge soft targets for the input kiwifruit images, and then generates soft labels from the knowledge soft targets, and calculates the student model loss to finally obtain hard targets. The hard targets are used to predict the kiwifruit images to obtain recognition boxes. The recognition boxes are counted, which is the number of kiwifruits.

6. The knowledge distillation kiwifruit visual counting method according to claim 1, characterized in that: The student model is trained using the kiwi image dataset to generate knowledge soft targets, which are then used to generate soft labels.

7. The knowledge distillation kiwifruit visual counting method according to claim 1, characterized in that: The processing process of the calibration attention unit is expressed as: ； in, The predicted feature map output by one of the calibrated attention units; are input features, and is the sigmoid activation function, and ⊙ represents point-by-point multiplication.

8. The knowledge distillation kiwifruit visual counting method according to claim 7, characterized in that: The fusion process of the SBA adaptive attention module is expressed as: ； in represents a 3×3 convolution, is the predicted feature map output by another calibrated attention unit, Concat represents the concatenation operation; Z represents the feature map output by the SBA adaptive attention module.

Images