A Microscopic Data Detection Method for Apple Disease Spores Based on Multimodal and Semi-Supervised Learning

Abstract

This application discloses a method for detecting apple disease spores using microscopic data based on multimodal and semi-supervised learning. The method includes: acquiring raw microscopic data of apple disease fungal spores, performing edge detection and texture enhancement to generate texture-enhanced data; extracting features from the raw microscopic data and texture-enhanced data using a dual-branch encoder, and fusing them using a cross-attention mechanism to obtain enhanced visual features; inputting the textual description information of the disease fungal spores into a text encoder for encoding to obtain global text features; aligning the enhanced visual features and global text features across modalities based on a multimodal object detection network, outputting multimodal fused features, and inputting them into a semi-supervised learning framework to train a student-teacher model using labeled and unlabeled data; inputting the microscopic data to be detected into the trained model and outputting the detection results. This method improves the detection accuracy and robustness of microscopic data with extremely low annotation costs.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent detection of agricultural diseases, specifically to a method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning. Background Technology

[0002] Microscopic data detection of apple disease spores is a crucial step in the early diagnosis and control of agricultural diseases. Traditional methods rely on manual microscopic observation, which is inefficient, subjective, and unsuitable for large-scale, real-time monitoring. Existing deep learning-based detection methods often employ a single visual modality, failing to fully utilize the microscopic texture features and semantic information of spore images, and exhibiting limited generalization ability when labeled data is scarce. In recent years, multimodal learning and semi-supervised learning have made significant progress in natural image detection. However, in the field of agricultural microscopic images, especially in the detection of apple disease spores, the following challenges remain: spores are tiny in morphology, with small inter-class differences, making it difficult to distinguish them using a single visual feature; the cost of acquiring labeled data is high, and a large amount of unlabeled data remains unutilized; complex backgrounds, lighting variations, and image noise interfere with detection accuracy.

[0003] Therefore, how to combine multimodal prior knowledge with semi-supervised learning mechanisms to achieve high-precision detection of apple disease spores with extremely low annotation ratios has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, this application proposes the following technical solution:

[0005] This application provides a method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning, including:

[0006] After obtaining the raw microscopic data of apple disease fungal spores, edge detection and texture enhancement processing are performed on the raw microscopic data to generate texture enhancement data;

[0007] Visual features of the original microscopic data and texture enhancement data are extracted by a dual-branch encoder, and cross-branch feature fusion is performed using a cross-attention mechanism to obtain enhanced visual features.

[0008] The textual description information of each type of apple disease fungal spore is input into a text encoder for encoding to obtain global text features;

[0009] Based on a multimodal object detection network, the enhanced visual features and the global text features are aligned and interacted across modalities to output multimodal fusion features;

[0010] The multimodal fusion features are input into the semi-supervised learning framework Soft-Teacher, and the student-teacher model is trained by combining labeled and unlabeled data.

[0011] Input the microscopic data of the apple disease spores to be detected into the trained student-teacher model, and output the detection results of the apple disease spores.

[0012] In one possible implementation, after obtaining the raw microscopic data of apple disease fungal spores, edge detection and texture enhancement processing are performed on the raw microscopic data to generate texture-enhanced data, including:

[0013] Obtain raw microscopic data of fungal spores that cause apple diseases;

[0014] The edge data of the original microscopic data were extracted using the Canny edge detection algorithm, and the calculation formula is as follows:

[0015] ;

[0016] The edge data is concatenated with the original microscopic data, and then input into a convolutional layer for texture feature enhancement to generate texture-enhanced data. The calculation formula is as follows:

[0017]

[0018] in, For raw microscopic data, For edge detection operators, For edge data, for Convolutional layer Enhance the texture data.

[0019] In one possible implementation, the visual features of the original microscopic data and texture enhancement data are extracted separately using a dual-branch encoder, and cross-branch feature fusion is performed using a cross-attention mechanism to obtain enhanced visual features, including:

[0020] The original microscopic data and texture enhancement data are respectively input into a dual-branch encoder. The first branch extracts local and global features from the original microscopic data, and the second branch extracts texture structure features. The calculation formulas are as follows:

[0021]

[0022]

[0023] in, These are local features of the original microscopic data. For raw microscopic data, The number of samples processed at one time. The number of feature channels, H,W These represent the height and width of the feature data, respectively. For texture enhancement data, For texture structure features, For the real number field;

[0024] By utilizing a cross-attention mechanism, the extracted texture structure features and the features of the original microscopic data are fused across branches to obtain enhanced visual features.

[0025] In one possible implementation, the method of using a cross-attention mechanism to perform cross-branch feature fusion of extracted texture structure features and features from the original microscopic data to obtain enhanced visual features includes:

[0026] The extracted texture structure features and the features of the original microscopic data are used to generate query vector Q, key vector K, and value vector V respectively through linear mapping. The calculation formula is as follows:

[0027]

[0028] By calculating attention weights, the most relevant contextual information to the structural regions indicated by the texture structure features is selected from the features of the original microscopic data. This information is then aggregated and added to the features of the original microscopic data via residual connections to obtain enhanced visual features. The calculation formula is as follows:

[0029]

[0030]

[0031] in, To enhance visual features, For layer normalization operation, for Attention weights between them This means converting similarity scores into attention weights. Let be the dimension of the key vector. This is a linear transformation operation.

[0032] In one possible implementation, the step of inputting the textual description information of each type of apple disease fungal spore into a text encoder for encoding to obtain global text features includes:

[0033] For each type of apple disease fungal spore, the category name is inserted into a predefined prompt template to generate a category-related text description. The calculation formula is as follows:

[0034]

[0035] The text description is input into the CLIP text encoder for encoding. The feature vector corresponding to the [CLS] marker in the encoded sequence is extracted as the global text feature. The calculation formula is as follows:

[0036]

[0037] ;

[0038] in, for The prompt text is composed of categories. For category name, The length of the text sequence. For feature dimension, The number of samples processed at one time. For the real number field, For text global features, For text encoders, Features output by the text encoder.

[0039] In one possible implementation, the multimodal object detection network performs cross-modal alignment and interaction between the enhanced visual features and the text global features to output multimodal fusion features, including:

[0040] The enhanced visual features are input into the visual encoder of Grounding DINO to extract multi-scale visual tokens. ;

[0041] The global text features are input into Grounding DINO's text encoder to extract the text token. ;

[0042] The multi-scale visual token With text token Visual-language alignment is performed on the input cross-modal decoder, and the data-text similarity matrix is ​​calculated using the following formula:

[0043] ;

[0044] in, For data-text similarity matrix, For data feature tensors, For text feature tensors, The number of samples processed at one time. The number of tokens for the data. The number of tokens for the text. For feature dimensions;

[0045] Based on the data-text similarity matrix, an object-text matching score is generated, and the matching score is combined to output a multimodal fusion feature containing the location and classification information of disease spores.

[0046] In one possible implementation, the multimodal fusion features are input into the semi-supervised learning framework Soft-Teacher, and the student-teacher model is trained by combining labeled and unlabeled data, including:

[0047] The teacher model is trained using labeled data while inference is performed on unlabeled data. Predictions with a confidence level higher than a preset threshold are used as pseudo-labeled data. The calculation formula is as follows:

[0048]

[0049] in, Unlabeled data For unlabeled data The generated pseudo-tags For the teacher model, The confidence level of the teacher model prediction. The confidence threshold;

[0050] The student model is trained using the labeled data and the pseudo-labeled data, while the teacher model is updated using the exponential moving average of the student model parameters. The calculation formula is as follows:

[0051]

[0052] in, For the parameters of the teacher model, For the parameters of the student model, This is the coefficient of the exponential moving average.

[0053] In one possible implementation, the teacher model is trained using labeled data, with the loss function being:

[0054]

[0055] in, This is a text description corresponding to the disease category. Label the actual bounding box. The bounding box coordinates predicted by the teacher model. For the supervised loss of the teacher model, For bounding box regression loss, For text-target matching loss.

[0056] In one possible implementation, the student model is trained based on the labeled data and the pseudo-labeled data, with the loss function being:

[0057]

[0058] in, For the unsupervised loss of the student model, For student models to predict strongly augmented unlabeled data, For indicator functions, For the bounding boxes predicted by the student model for the same unlabeled data, For the text label corresponding to the pseudo-boundary box, The confidence level of the teacher model's prediction of the pseudo-boundary box. The confidence threshold. For bounding box regression loss, For text-target matching loss.

[0059] In one possible implementation, a lightweight fine-tuning module is used during training, with the following structure:

[0060] Insert an Adapter module into each Transformer layer. The structure is dimensionality reduction-activation-dimensionality increase, and the calculation formula is:

[0061]

[0062] Then, it is fused with the original features through residual connections, and the calculation formula is as follows:

[0063]

[0064] in, Input the original features. For the final feature output, For learnable scaling factor, For a dimension reduction matrix, For an upgraded matrix, As a bottleneck dimension, For feature dimension, For activation function, Output for the adapter module.

[0065] Compared with the prior art, the beneficial effects of this application are as follows:

[0066] This application significantly improves the detection accuracy and robustness of apple disease spore microscopic data with extremely low annotation costs by integrating multimodal prior knowledge, texture enhancement and semi-supervised learning mechanisms, providing a feasible technical solution for intelligent diagnosis of agricultural diseases. Attached Figure Description

[0067] Figure 1 A flowchart illustrating a method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning, provided in an embodiment of this application;

[0068] Figure 2 A data acquisition flowchart provided for embodiments of this application;

[0069] Figure 3 A partial dataset illustration provided for embodiments of this application;

[0070] Figure 4 This is a diagram of a dual-branch grounding-dino structure provided in an embodiment of this application;

[0071] Figure 5 A semi-supervised learning framework diagram provided for embodiments of this application;

[0072] Figure 6 The detection frame and confidence level provided in the embodiments of this application;

[0073] Figure 7 The image shows the ablation experiment results provided in the embodiments of this application. Detailed Implementation

[0074] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.

[0075] Figure 1 A flowchart illustrating a method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning, provided in this application embodiment, is shown below. Figure 1 This embodiment presents a method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning, comprising:

[0076] S101: After obtaining the original microscopic data of apple disease fungal spores, edge detection and texture enhancement processing are performed on the original microscopic data to generate texture enhancement data.

[0077] See Figure 2In this embodiment, the raw microscopic data of apple disease fungal spores uses microscopic images of apple disease fungal spores. A self-constructed dataset of apple disease fungal spore microscopic images is used, containing spores from three common diseases: apple brown spot fungus, apple rust fungus, and apple rot fungus. The images consist of two parts: one part is images taken under a microscope under the same environmental conditions after laboratory sectioning of diseased leaves; the other part is collected using a spore analyzer, totaling 600 images. To improve model robustness and conform to the semi-supervised training paradigm, weak and strong data augmentation operations were performed on the training images. Weak data augmentation included random horizontal and vertical flipping and random rotation, while strong data augmentation included random adjustment of brightness and contrast and Gaussian noise injection. Furthermore, to obtain accurate detection data, the minimum bounding rectangle of the spores was used as the ground truth for manual image annotation, reducing interference from background information in the actual boxes. The spores of *Botrytis cinerea* were classified as belonging to the genus *Botrytis*. The background was unmarked. Furthermore, to verify the performance of semi-supervised learning, only 1%, 5%, and 10% of the images were labeled using the Labelme annotation dataset. Finally, an XML file containing object type and coordinate information was saved, such as... Figure 3 As shown, standard brown spot spores are dicellous, oblong, and slightly pointed at both ends; rot spores are unicellular and sausage-shaped; and rust spores are dicellous, brown, oblong, and have long stalks. Due to the influence of light, spore extraction procedures, and microscopy, the task of spore detection faces the following significant challenges.

[0078] Subsequently, the Canny edge detection algorithm was used to extract the edge map of the original microscopic image. The calculation formula is as follows:

[0079] ;

[0080] The edge map is concatenated with the original microscopic image through channels, and then input into a convolutional layer for texture feature enhancement to generate a texture-enhanced image. The calculation formula is as follows:

[0081]

[0082] in, The original microscopic image, For edge detection operators, For edge maps, for Convolutional layers are used for smoothing and feature fusion. Enhance the texture of the image.

[0083] S102 extracts visual features from the original microscopic image and the texture-enhanced image through a dual-branch image encoder, and uses a cross-attention mechanism to fuse cross-branch features to obtain enhanced visual features.

[0084] In this embodiment, to enable the model to distinguish key morphological features of tiny targets such as spores, a dual-branch coding architecture is used to simultaneously extract and fuse global semantic information and local structural information of the image, such as... Figure 4 As shown, the dual-branch image encoder in this embodiment includes: a first branch, a CLIP-pretrained visual encoder, used to extract local and global features from the original spore image; and a second branch, a visual encoder with the same structure, whose input is the spore image after edge detection and texture enhancement, used to extract texture structure features. The original microscopic image is input to the CLIP-pretrained ResNet-50 encoder to extract local and global features from the original microscopic image, and the texture-enhanced image is input to ConvNeXt. This encoder can learn and abstract deep feature representations related to complex texture patterns from simple contour information. Let the function of this encoder be... ,in If the parameters are learnable, then the output deep texture features The calculation formulas are as follows:

[0085]

[0086]

[0087] in, These are local features of the original microscopic image. The original microscopic image, The number of samples processed at one time. The number of feature channels, H,W These represent the height and width of the feature map, respectively. To enhance the texture of the image, For texture structure features, It is the field of real numbers.

[0088] In order to extract deep texture structure features To effectively fuse features with those of the original microscopic image, this embodiment designs a lightweight cross-attention fusion layer. In this layer, texture structure features are... Reshape into query vector The features of the original microscopic image are simultaneously used as key vectors. Sum value vector By calculating attention weights, the model can dynamically select the most relevant contextual information from the features of the original microscopic image that is most relevant to the structural regions indicated by the texture structure features. This information is then aggregated and added to the features of the original microscopic image via residual connections to obtain enhanced visual features. The calculation formula is as follows:

[0089]

[0090]

[0091]

[0092] in, Let be the dimension of the key vector. For linear transformation operations, For layer normalization operation, To enhance visual features, This means that similarity scores are converted into attention weights. This texture-guided fusion mechanism allows the model to focus on the edge and shape priors of spores at the feature level, thereby significantly improving the model's localization accuracy for small targets and its robustness in complex backgrounds.

[0093] S103, input the text description information of each type of apple disease fungal spore into the text encoder for encoding to obtain the global text features.

[0094] In this embodiment, for each type of apple disease fungal spore, the category name is inserted into a predefined prompt template to generate a category-related text description. The calculation formula is as follows:

[0095]

[0096] The text description is input into the CLIP text encoder for encoding. The feature vector corresponding to the [CLS] marker in the encoded sequence is extracted as the global text feature. This feature will be used for subsequent multimodal alignment and detection tasks. The calculation formula is as follows:

[0097]

[0098] ;

[0099] in, for The prompt text is composed of categories. For category name, The length of the text sequence. For feature dimension, For text global features, For text encoders, For the real number field, The number of samples processed at one time. Features output by the text encoder.

[0100] S104, based on a multimodal object detection network, performs cross-modal alignment and interaction between enhanced visual features and global text features, and outputs multimodal fusion features.

[0101] In this embodiment, enhanced visual features are input into the visual encoder of Grounding DINO to extract multi-scale visual tokens. The text global features are input into the Grounding DINO text encoder to extract text tokens. Multi-scale visual tokens With text token Visual-language alignment is performed on the input cross-modal decoder, and the image-text similarity matrix is ​​calculated using the following formula:

[0102]

[0103] in, For image-text similarity matrix, For image feature tensors, For text feature tensors, The number of samples processed at one time. The number of tokens for the image. The number of tokens for the text. The feature dimension is defined as follows: Object-text matching scores are generated based on the image-text similarity matrix, and the matching scores are combined to output multimodal fusion features containing disease spore location and classification information.

[0104] S105. The multimodal fusion features are input into the semi-supervised learning framework Soft-Teacher, and the student-teacher model is trained by combining labeled and unlabeled data.

[0105] To address the dual limitations of difficult data labeling and fixed detection categories, this embodiment designs a semi-supervised open-vocabulary collaborative training mechanism. This mechanism organically combines a semi-supervised learning framework based on consistency regularization with an open-vocabulary visual localization model, enabling the model to learn general representations from large amounts of unlabeled data while maintaining its ability to flexibly define detection targets using natural language. This mechanism constructs a student-teacher dual-model architecture, such as... Figure 5 As shown, the student model and the teacher model have the exact same structure, but their parameter update methods differ. In each training iteration, labeled and unlabeled data batches are mixed and sampled.

[0106] Use labeled data The teacher model is trained using the following loss function:

[0107]

[0108] in, Microscopic images of tagged spores. This is a text description corresponding to the disease category. Label the actual bounding box. The bounding box coordinates predicted by the teacher model. For the supervised loss of the teacher model, For bounding box regression loss, For text-target matching loss.

[0109] The teacher model performs inference on unlabeled data. First, a set of weak data augmentations, such as random flipping and scaling, are applied to the input image and then fed into the teacher model to generate predictions. The parameters of the teacher model... These are student model parameters. The historical exponential moving average, i.e. This makes its predictions more stable than those of the student network. The teacher model is updated using an exponential moving average of the student model parameters, calculated as follows:

[0110]

[0111] in, For the parameters of the teacher model, For the parameters of the student model, The coefficients are used for the exponential moving average. A threshold is then applied to filter the predicted confidence scores, retaining high-confidence predictions as pseudo-labels. Next, a stronger set of data augmentations, such as color jittering and random erasure, is applied to the same batch of unlabeled images and fed into the student model. Unsupervised loss. The model is constructed by calculating the consistency between the student model's predictions on strongly enhanced images and the pseudo-labels generated by the teacher model. The calculation formula is as follows:

[0112]

[0113] in, For the unsupervised loss of the student model, For student models to predict strongly augmented unlabeled data, For indicator functions, For the bounding boxes predicted by the student model for the same unlabeled data, For the text label corresponding to the pseudo-boundary box, The confidence level of the teacher model's prediction of the pseudo-boundary box. The confidence threshold. For bounding box regression loss, For text-to-target matching loss, the total training loss is a weighted sum of two terms: , This represents the unsupervised loss weight.

[0114] This architecture achieves synergy through two key design features. First, the teacher model's EMA updates and pseudo-label generation provide a continuous and stable learning signal for the student model on unlabeled data. Second, the consistency of strong and weak data enhancement forces the student model to learn more essential feature representations that are invariant to irrelevant noise. Ultimately, the trained student model not only possesses the ability to learn from limited annotations, but more importantly, its internal vision-language alignment module enables it to directly accept and respond to new categories of text descriptions not encountered during training, achieving open-vocabulary detection and thus meeting the urgent need for rapid response to emerging diseases in agricultural practice.

[0115] In addition, the Adapter module is used for lightweight fine-tuning during training, with the following structure:

[0116] Insert an Adapter module into each Transformer layer. The structure is dimensionality reduction-activation-dimensionality increase, and the calculation formula is:

[0117]

[0118] Then, it is fused with the original features through residual connections, and the calculation formula is as follows:

[0119]

[0120] in, Input the original features. For the final feature output, For learnable scaling factor, For a dimension reduction matrix, For an upgraded matrix, As a bottleneck dimension, For feature dimension, For activation function, Output for the adapter module.

[0121] S106: Input the microscopic image of the apple disease spores to be detected into the trained student-teacher model, and output the detection results of the apple disease spores.

[0122] See Figure 6 In this embodiment, the microscopic image of the apple disease spores to be detected is input into the trained student-teacher model to output the bounding box position, disease category and confidence level, supporting real-time detection and disease statistics.

[0123] like Figure 7 As shown, this embodiment uses average precision (mAP), average recall (mAR), F1 score, inference speed (FPS), and floating-point operations (GFloaps) to evaluate model performance. The formula for calculating average precision is:

[0124]

[0125] in, The total number of categories, For the first In terms of average accuracy, this method achieves 91.6% mAP with only 1% labeled data, 97.5% mAP with 5% labeled data, and 97.5% mAP with only 10% labeled data.

[0126] This application effectively improves the accuracy and practicality of apple disease spore detection through the synergy of multimodal guidance, texture enhancement, and semi-supervised learning, making it suitable for intelligent monitoring scenarios of agricultural diseases.

[0127] This application's embodiments construct an efficient and low-cost apple disease spore detection system by integrating texture enhancement, multimodal alignment, and semi-supervised learning, suitable for intelligent monitoring and early warning scenarios of agricultural diseases. This method significantly reduces annotation costs while ensuring detection accuracy, demonstrating good practicality and promotional value.

[0128] In this embodiment, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0129] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0130] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for detecting apple disease spore microscopic data based on multi-modal and semi-supervised learning, characterized in that, include: After obtaining the raw microscopic data of apple disease fungal spores, edge detection and texture enhancement processing are performed on the raw microscopic data to generate texture enhancement data; Visual features are extracted from the original microscopic data and texture enhancement data using a dual-branch encoder, and cross-branch feature fusion is performed using a cross-attention mechanism to obtain enhanced visual features, including: The original microscopic data and texture enhancement data are respectively input into a dual-branch encoder. The first branch extracts local and global features from the original microscopic data, and the second branch extracts texture structure features. The calculation formulas are as follows: wherein, is a local feature of the original microscopic data, is the original microscopic data, is a number of samples processed at a time, is a number of feature channels, H,W are height and width of the feature data, respectively, is texture enhanced data, is a texture structure feature, is a real number field; By utilizing a cross-attention mechanism, extracted texture structure features and features from the original microscopic data are fused across branches to obtain enhanced visual features, including: The extracted texture structure features and the features of the original microscopic data are used to generate query vector Q, key vector K, and value vector V respectively through linear mapping. The calculation formula is as follows: By calculating attention weights, the most relevant contextual information to the structural regions indicated by the texture structure features is selected from the features of the original microscopic data. This information is then aggregated and added to the features of the original microscopic data via residual connections to obtain enhanced visual features. The calculation formula is as follows: in, To enhance visual features, For layer normalization operation, for Attention weights between them This means converting similarity scores into attention weights. Let be the dimension of the key vector. This is a linear transformation operation; The textual description information of each type of apple disease fungal spore is input into a text encoder for encoding to obtain global text features; Based on a multimodal object detection network, the enhanced visual features and the global text features are aligned and interacted across modalities to output multimodal fusion features; The multimodal fusion features are input into the semi-supervised learning framework Soft-Teacher, and the student-teacher model is trained by combining labeled and unlabeled data. Input the microscopic data of the apple disease spores to be detected into the trained student-teacher model, and output the detection results of the apple disease spores.

2. The method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning according to claim 1, characterized in that, After obtaining the raw microscopic data of apple disease fungal spores, edge detection and texture enhancement processing are performed on the raw microscopic data to generate texture-enhanced data, including: Obtain raw microscopic data of fungal spores that cause apple diseases; The edge data of the original microscopic data were extracted using the Canny edge detection algorithm, and the calculation formula is as follows: ； The edge data is concatenated with the original microscopic data, and then input into a convolutional layer for texture feature enhancement to generate texture-enhanced data. The calculation formula is as follows: in, For raw microscopic data, For edge detection operators, For edge data, for Convolutional layer Enhance the texture data.

3. The method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning according to claim 1, characterized in that, The text description information of each type of apple disease fungal spore is input into a text encoder for encoding to obtain global text features, including: For each type of apple disease fungal spore, the category name is inserted into a predefined prompt template to generate a category-related text description. The calculation formula is as follows: The text description is input into the CLIP text encoder for encoding. The feature vector corresponding to the [CLS] marker in the encoded sequence is extracted as the global text feature. The calculation formula is as follows: ； in, for The prompt text is composed of categories. For category name, The length of the text sequence. For feature dimension, The number of samples processed at one time. For the real number field, For text global features, For text encoders, Features output by the text encoder.

4. The method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning according to claim 1, characterized in that, The multimodal object detection network performs cross-modal alignment and interaction between the enhanced visual features and the global text features, outputting multimodal fusion features, including: The enhanced visual features are input into the visual encoder of Grounding DINO to extract multi-scale visual tokens. ; The global text features are input into Grounding DINO's text encoder to extract the text token. ; The multi-scale visual token With text token Visual-language alignment is performed on the input cross-modal decoder, and the data-text similarity matrix is ​​calculated using the following formula: ； in, For data-text similarity matrix, For data feature tensors, For text feature tensors, The number of samples processed at one time. The number of tokens for the data. The number of tokens for the text. For feature dimensions; Based on the data-text similarity matrix, an object-text matching score is generated, and the matching score is combined to output a multimodal fusion feature containing the location and classification information of disease spores.

5. The method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning according to claim 1, characterized in that, The multimodal fusion features are input into the semi-supervised learning framework Soft-Teacher, and the student-teacher model is trained by combining labeled and unlabeled data, including: The teacher model is trained using labeled data while inference is performed on unlabeled data. Predictions with a confidence level higher than a preset threshold are used as pseudo-labeled data. The calculation formula is as follows: in, Unlabeled data For unlabeled data The generated pseudo-tags For the teacher model, The confidence level of the teacher model prediction. The confidence threshold; The student model is trained using the labeled data and the pseudo-labeled data, while the teacher model is updated using the exponential moving average of the student model parameters. The calculation formula is as follows: in, For the parameters of the teacher model, For the parameters of the student model, This is the coefficient of the exponential moving average.

6. The method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning according to claim 5, characterized in that, The teacher model is trained using labeled data, with the loss function being: in, This is a text description corresponding to the disease category. Label the true bounding box. The bounding box coordinates predicted by the teacher model. For the supervised loss of the teacher model, For bounding box regression loss, For text-target matching loss.

7. The method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning according to claim 5, characterized in that, The student model is trained using the labeled data and the pseudo-labeled data, with the loss function being: in, For the unsupervised loss of the student model, For student models to predict strongly augmented unlabeled data, For indicator functions, For the bounding boxes predicted by the student model for the same unlabeled data, For the text label corresponding to the pseudo-boundary box, The confidence level of the teacher model's prediction of the pseudo-boundary box. The confidence threshold. For bounding box regression loss, For text-target matching loss.

8. The method for detecting apple disease spore microscopic data based on multimodal and semi-supervised learning according to claim 5, characterized in that, Lightweight fine-tuning is performed using the Adapter module during training, with the following structure: Insert an Adapter module into each Transformer layer. The structure is dimensionality reduction-activation-dimensionality increase, and the calculation formula is: Then, it is fused with the original features through residual connections, and the calculation formula is as follows: in, Input the original features. For the final feature output, For learnable scaling factor, For a dimension reduction matrix, For an upgraded matrix, As a bottleneck dimension, For feature dimension, For activation function, Output for the adapter module.

Images