Collaborative learning-based crown classification method and apparatus, and storage medium

Abstract

The invention provides a tree crown classification method and device based on collaborative learning and a storage medium, and relates to the technical field of tree crown recognition. According to the method, pre-training and updating training are respectively performed on the Teaser network and the Student network, then appropriate pseudo samples are screened through a small-amplitude multi-order pseudo sample sampling mechanism, and the screened pseudo samples are trained by adopting a deep collaborative learning strategy progressive exchange learning mode, so that the crown texture and the spatial topological structure are more comprehensively perceived, and the tree crown texture and the spatial topological structure are more comprehensively perceived. And the multi-type crown identification precision is improved under the constraint of small samples.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of tree crown recognition, and specifically relates to a tree crown classification method, device and storage medium based on collaborative learning. Background Art

[0002] The identification of tree crowns in forests is of great value for judging tree growth, predicting biomass, controlling pests and diseases, and protecting biodiversity, and is of great significance to forest resource management. However, existing research focuses on obtaining information on single-type tree crowns, and relies heavily on a large number of labeled remote sensing tree crown image samples, which has high application costs and extracts few forest area parameters.

[0003] In this regard, the existing technology uses semi-supervised learning methods for tree crown recognition, but it needs to introduce external data such as Lidar. The labeling and processing of these data still requires a lot of cost, and it is easy to introduce noise, so the applicability is not high. In addition, existing research mainly focuses on single tree crown recognition, which reduces the effectiveness of forest information extraction. In this regard, the "teacher-student" model is used to carry out semi-supervised target detection to cope with the challenges faced. However, in most detection methods, each round of interactive learning is a one-time improvement or screening of the entire set of pseudo samples. When there are very few labeled samples, it will lead to a large number of newly acquired pseudo samples of low quality, and the degree of network capacity improvement is also extremely limited. When the image has strong local information but weak overall information, it will produce a large number of low-quality pseudo samples that are extremely unfavorable for model training. It can be seen that exploring a large amount of unlabeled data to obtain application-level tree crown detectors is an important challenge currently faced. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a tree crown classification method, device and storage medium based on collaborative learning in view of the deficiencies in the prior art.

[0005] The technical solution of the present invention to solve the above technical problem is as follows: a tree crown classification method based on collaborative learning, comprising the following steps:

[0006] S1. constructing a training tree crown image set containing tree crown images, wherein the training tree crown image set includes a plurality of labeled training tree crown images and a plurality of unlabeled training tree crown images;

[0007] S2. Build an initial Teacher network and an initial Student network with the same configuration based on the FT-Transformer architecture;

[0008] S3, in the pre-training stage, pre-training the initial Teacher network and the initial Student network through a plurality of the labeled training tree crown images to obtain a Teacher network and a Student network;

[0009] S4, in the semi-supervised learning stage, a plurality of the unlabeled training tree crown images are input into the Teacher network to generate an initial pseudo sample set, the parameters of the Student network are updated by the initial pseudo sample set to obtain a new Student network, and the parameters of the new Student network are synchronously updated to the Teacher network to obtain a new Teacher network;

[0010] S5. In the collaborative learning stage, a collaborative training data set is constructed from a plurality of the labeled training tree crown images and the initial pseudo sample set, pseudo samples are screened for the collaborative training data set through the new Teacher network to generate an optimized pseudo sample set, the collaborative training data set is optimized and trained through the new Student network to obtain an optimized Student network, and the parameters of the optimized Student network are updated to the new Teacher network;

[0011] S6, repeat S5 until the preset number of collaborative training times is reached to obtain the optimal Teacher network;

[0012] S7, importing the tree crown image to be classified into the optimal Teacher network, and outputting the tree crown type result through the optimal Teacher network.

[0013] Another technical solution of the present invention to solve the above technical problem is as follows: A tree crown classification device based on collaborative learning, comprising:

[0014] A crown image set module, used to construct a training crown image set containing crown images, wherein the training crown image set includes a plurality of labeled training crown images and a plurality of unlabeled training crown images;

[0015] The initial network building module is used to build the initial Teacher network and the initial Student network with the same configuration based on the FT-Transformer architecture;

[0016] A pre-training module, used for pre-training the initial Teacher network and the initial Student network by using a plurality of the labeled training tree crown images to obtain a Teacher network and a Student network during the pre-training phase;

[0017] An update training module is used to input a plurality of the unlabeled training tree crown images into the Teacher network in the semi-supervised learning stage to generate an initial pseudo sample set, update the parameters of the Student network through the initial pseudo sample set to obtain a new Student network, and synchronously update the parameters of the new Student network to the Teacher network to obtain a new Teacher network;

[0018] An optimization module is used to construct a collaborative training data set from a plurality of the labeled training tree crown images and the initial pseudo sample set during the collaborative learning stage, perform pseudo sample screening on the collaborative training data set through the new Teacher network to generate an optimized pseudo sample set, perform optimization training on the collaborative training data set through the new Student network to obtain an optimized Student network, update the parameters of the optimized Student network to the new Teacher network, and perform training in a loop until a preset number of collaborative training times is reached to obtain an optimal Teacher network;

[0019] The classification module is used to import the tree crown image to be classified into the optimal Teacher network, and output the tree crown type result through the optimal Teacher network.

[0020] Another technical solution of the present invention to solve the above-mentioned technical problem is as follows: a tree crown classification device based on collaborative learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the tree crown classification method based on collaborative learning as described is implemented.

[0021] Another technical solution of the present invention to solve the above technical problem is as follows: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the tree crown classification method based on collaborative learning as described is implemented.

[0022] The beneficial effects of the present invention are: through multiple rounds of deep collaborative learning, a small-scale pseudo sample set is used in multi-frequency dual-model interaction to continuously improve the model capability, and a small-scale multi-order pseudo sample sampling mechanism and a corresponding model interaction feedback mode are adopted to improve the recognition accuracy of multiple types of tree crowns under small sample constraints. Pseudo sample screening indicators are designed to improve the quality of pseudo sample sets, and the knowledge contained in unlabeled data can be more fully captured and applied to achieve asymptotic enhancement of the model. Secondly, a basic network model with multi-type attention is constructed based on the FT-Transformer architecture to more comprehensively perceive the tree crown texture and spatial topological structure, improve the degree of tree crown recognition, and improve the effect of collaborative learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a tree crown classification method based on collaborative learning provided by an embodiment of the present invention;

[0024] Figure 2 The overall framework diagram of semi-supervised active learning provided by the embodiment of the present invention;

[0025] Figure 3 The overall architecture diagram of the FT-DETR framework provided by the embodiment of the present invention;

[0026] Figure 4 A module structure diagram of the Backbone subnet provided in an embodiment of the present invention;

[0027] Figure 5 A module block diagram of a tree crown classification device based on collaborative learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0029] like Figure 1 and Figure 2 As shown, a tree crown classification method based on collaborative learning provided by an embodiment of the present invention includes the following steps:

[0030] S1. Constructing a training tree crown image set containing tree crown images, wherein the training tree crown image set includes a plurality of labeled training tree crown images and a plurality of unlabeled training tree crown images.

[0031] S2. Build an initial Teacher network and an initial Student network with the same configuration based on the FT-Transformer architecture.

[0032] S3. In the pre-training stage, the initial Teacher network and the initial Student network are pre-trained using a plurality of the labeled training tree crown images to obtain the Teacher network and the Student network. Specifically, in the pre-training stage, the initial Teacher network and the initial Student network are supervisedly learned using all the labeled data to obtain the initial network parameters of the two networks.

[0033] S4. In the semi-supervised learning stage, a plurality of the unlabeled training tree crown images are input into the Teacher network to generate an initial pseudo-sample set, the parameters of the Student network are updated through the initial pseudo-sample set to obtain a new Student network, and the parameters of the new Student network are synchronously updated to the Teacher network to obtain a new Teacher network. Specifically, in the semi-supervised learning stage, all unlabeled data are used to generate an initial pseudo-sample set based on the Teacher network, and then the Student network is trained using the initial pseudo-sample set, and then all parameters of the Teacher network are synchronously updated. In other words, the unlabeled data is labeled by the pre-trained Teacher network to obtain initial pseudo-labeled data, and the Student network is used to learn more pseudo-sample labeled data to improve the recognition accuracy of its own network, and the Teacher network is updated with the improved network parameters so that the Teacher network can label more tree crown types.

[0034] S5. In the collaborative learning stage, a collaborative training data set is constructed from a plurality of the labeled training crown images and the initial pseudo-sample set. The collaborative training data set is screened for pseudo-samples through the new Teacher network to generate an optimized pseudo-sample set. The collaborative training data set is optimized and trained through the new Student network to obtain an optimized Student network, and the parameters of the optimized Student network are updated to the new Teacher network. Specifically, in the collaborative learning stage, a number of pseudo-samples are randomly selected from the initial pseudo-sample set and merged with the pseudo-samples obtained in the previous round of learning to form a data set for this round of learning; the above pseudo-sample sets are applied to the new Teacher network and the new Student network respectively, wherein the new Teacher network is responsible for generating new pseudo-samples; the new Student network optimizes its own parameters through supervised learning; the parameters of the new Student network are updated to the new Teacher network to obtain a new round of new Teacher network.

[0035] S6, repeat S5 until the preset number of collaborative training times is reached to obtain the optimal Teacher network. Specifically, repeat the steps of the collaborative learning stage until active learning is fully completed; and take the Teacher model with the highest verification performance as the final available model.

[0036] S7, importing the tree crown image to be classified into the optimal Teacher network, and outputting the tree crown type result through the optimal Teacher network.

[0037] In an embodiment of the present invention, a small-amplitude multi-order pseudo-sample sampling mechanism and a corresponding model interaction feedback mode are adopted, and a new pseudo-sample information force measurement index is designed, which can more fully capture and apply the knowledge contained in the unlabeled data to achieve the asymptotic enhancement of the model; secondly, a base model with multi-type attention is introduced to more comprehensively perceive the crown texture and spatial topological structure, improve the degree of crown recognition, and improve the effect of collaborative learning.

[0038] Preferably, S2 is specifically:

[0039] like Figure 3 The FT-Transformer shown in the figure introduces a more complex "head" subnet in the DETR model architecture to build an initial Teacher network and an initial Student network with the same configuration. The overall architecture of FT-DETR can be divided into a Partition subnet, a Backbone subnet, an Encoder encoding subnet, a Decoder decoding subnet, and a Neck prediction subnet.

[0040] The Partition subnet divides the input remote sensing image into 4x4 patches and flattens them in the band direction; then, the linear embedding layer is used to perform a linear transformation on the band data of each pixel.

[0041] The Backbone subnet is the backbone feature extraction network, which carries a large part of the computational cost. It can efficiently process large-size and high-resolution images and determines the feature representation capability. It is mainly composed of 6 Swin Transformer Blocks (STBs) connected in series. Each STB includes 1 Patch Merging module (but the first STB does not include a submodule) and 2 multi-head attention modules, such as Figure 4 As shown in the figure, Patch Merging is used to downsample the input feature map (reduce the resolution); the multi-head attention module is used to model the global spatial dependency between upper-layer features. Each STB uses the window multi-head attention (W-MSA) to capture the spatial dependency within the block, and then uses the window drift multi-head attention mechanism (SW-MSA) to capture the spatial dependency between blocks.

[0042] The encoder encoding subnet is used to strengthen the backbone features, which is mainly composed of dilated attention (DA) and FPN. The input feature layer of the encoder comes from the output of the 4th, 5th, and 6th STBs respectively. They flow into the DA to fuse the initial features of different levels of detail as needed, and then are sent to the Transformer module in combination with the position information to obtain strong semantic information. Based on CA (Coordinate Attention), DA uses dilated convolution and adaptive pooling layers to learn different receptive fields in each feature map, taking into account all spatial information while efficiently modeling the spatial and channel correlation relationship of the input features, reducing the loss of contextual information in feature channels and feature maps, and is more suitable for deeper and more comprehensive extraction of effective crown features under different lighting and shadows.

[0043] The decoder subnet is used to aggregate low-level physical features with high-level semantic features. It uses the Transformer module and feature query module (object queries) to process the enhanced feature layer to provide support for predicting multiple target classes at the same time. It uses a learnable vector containing N token feature representations to query the input feature layer to decode N corresponding possible target objects in parallel at a time; then the Transformer outputs N tokens after attention and mapping, and then sends them to Neck at the same time.

[0044] The neck prediction subnet consists of a classifier and a regressor combined with several fully connected layers (FFN). It processes the N tokens output by the decoder at one time and predicts the final detection result based on the multi-level features of the neck assembly. Each FFN consists of 2 linear layers with ReLU activation function. One FNN outputs the normalized center coordinates, height and width of the prediction box, and the other obtains the label of the corresponding prediction class.

[0045] The model loss function uses the Hungarian algorithm for bipartite graph matching, that is, a one-to-one comparison of the elements of the predicted set and the true set to minimize the matching loss:

[0046]

[0047] in, is the minimum matching loss that needs to be calculated, N is the actual number of objects, is the matching loss between the actual object i and a predicted object, y i For actual objects, is a predicted object that matches the actual object i, is the probability that the predicted object matched with the i-th actual object belongs to category c, ci is the category of the predicted object that matches the i-th actual object, b i is the i-th actual object bounding box, is the bounding box of the predicted object that matches the i-th actual object, which is the tree crown in the image.

[0048] In the embodiment of the present invention, the texture of different tree species in remote sensing images and the spatial dependency between tree crowns at different distances are very important for tree crown recognition. The earlier these shallow but important features are mined in the network model, the more useful it is to use the back end of the network to grasp the tree crown type and its range. Therefore, FT-Transformer specifically introduces a more complex "head" subnet on the DETR model architecture to obtain as many shallow key features of the tree crown as possible, and then uses a relatively simple network structure with an efficient attention module as the second half of the model to more accurately obtain effective deep features and improve the effect and efficiency of tree crown recognition.

[0049] Preferably, the S3 is specifically:

[0050] The initial Teacher network is used to learn the plurality of labeled training tree crown images to obtain a Teacher network, and all parameters of the Teacher network are input into the initial Student network to obtain a Student network.

[0051] In the embodiment of the present invention, the initial Teacher network is trained by a small amount of accurate labeled data, and then the trained parameters are input into the initial Student network, so that the Teacher network and the Student network with the same configuration have the same network parameters.

[0052] Preferably, the step of obtaining a new Student network in S4 further includes:

[0053] The parameters of the Student network are optimized by using the multiple labeled training tree crown images and the initial pseudo sample set, specifically:

[0054] The optimization loss function of the Student network is defined as:

[0055]

[0056] in, and They represent the supervised learning loss of the Student network and the unsupervised learning loss of the Teacher network, respectively, and γ is and Hyperparameters that trade off between

[0057] Supervised Learning Loss Function It is composed of classification losses and bounding box regressions of a plurality of the labeled training tree crown images, wherein the classification losses and bounding box regressions of the plurality of the labeled training tree crown images are obtained through labels of the labeled training tree crown images;

[0058] Unsupervised learning loss It is composed of the classification loss of the initial pseudo sample set, and the classification loss of the initial pseudo sample set is obtained by the pseudo labels of the initial pseudo sample set.

[0059] Specifically, By classification loss and bounding box regression The classification loss consists of:

[0060]

[0061] for:

[0062]

[0063] Among them, x l Refers to the labeled crown image sample, y cls is the label of the crown image, y loc is the location (bounding box) of the tree crown image, N l Represents x l The number of c Output tree crown image The c-th coordinate of is the c-th coordinate of the tree crown image.

[0064] for Express bounding box regression using a smooth L-1 loss:

[0065]

[0066] By classification loss composition:

[0067]

[0068] for:

[0069]

[0070] in, Refers to the labeled crown image samples, Pseudo labels for pseudo samples generated by the Teacher network, is the category of the tree crown image, N uis the total number of tree crown images.

[0071] In addition, in order to improve the quality of pseudo samples, the non-maximum suppression method (NMS) and the confidence threshold ρ are used to filter out repeated and uncertain pseudo samples. When the Student network is trained, the trained Student network is used to update low-quality pseudo samples (i.e., pseudo samples filtered out when the Teacher network screens pseudo samples), thereby converting all unlabeled data into pseudo sample data.

[0072] In the embodiment of the present invention, semi-supervised learning uses two detection networks with the same configuration, where the Teacher network is responsible for generating pseudo samples, and the Student network uses both pseudo samples and labeled images to optimize its own parameters to maximize model performance.

[0073] Collaborative learning involves the joint optimization and pseudo sample generation of the Teacher network and the Student network. In each round of collaborative learning, the Teacher network uses a small batch of pseudo samples obtained in the previous round. Generate a small batch of pseudo samples with higher information than the previous round As the training data of the Student network in the next round of learning; at the same time, based on The loss functions of the Student network and the Teacher network in this round are jointly optimized to update the Teacher network in the next round.

[0074] Preferably, the S5 comprises:

[0075] S501, constructing a collaborative training data set from the plurality of labeled training tree crown images and the initial pseudo sample set, wherein the collaborative training data set includes a recovered set, a screened set, and a rotation training set;

[0076] S502, extracting the collected collaborative recovery data according to preset parameters, obtaining the collaborative screening data of the previous round of collaborative training in the screening set, and merging the collaborative recovery data and the collaborative screening data into a round training set;

[0077] S503, performing weak data enhancement processing on the training set through the new Teacher network, and performing pseudo sample screening on the enhanced training set to generate a new pseudo sample set, performing strong data enhancement processing on the training set using the new Student network, and learning the enhanced training set to obtain an optimized Student network;

[0078] S504, screening the new pseudo sample set according to the pseudo sample screening strategy to obtain optimized pseudo samples;

[0079] S505, updating the parameters of the optimized Student network to the new Teacher network through the exponential moving average algorithm EMA, specifically:

[0080]

[0081] Among them, θ t and θ s are the parameters of the new Teacher network and the optimized Student network respectively, i is the i-th training stage, and β is the hyperparameter that determines the parameter transfer speed.

[0082] The S501 includes:

[0083] Constructing a first collection from a plurality of the labeled training tree crown images and the initial pseudo sample set;

[0084] When the collaborative screening data in the training set has been learned by the new Student network, the corresponding sample data is deleted from the training set according to the image information of the collaborative screening data and stored in the next collection;

[0085] When the number of the collaborative screening data in the round-robin set exceeds the preset round-robin number, the corresponding sample data is deleted from the round-robin set according to the image information of the collaborative screening data and stored in the next collection;

[0086] The new pseudo samples screened out according to the pseudo sample screening strategy are stored in the next collection;

[0087] The optimized pseudo samples are used as collaborative screening data for the current round of collaborative training and stored in the screening set.

[0088] Specifically, each round of learning involves the continuous change of three types of data sets storing pseudo samples, namely, the collection Filter Set and training set

[0089] Build collection Recycling Collection Used to provide new pseudo samples for each round of active learning (i.e. collaborative learning) and collect It is used to load the collection of pseudo samples filtered out (or actively deleted) in each round of active learning. Randomly extract pseudo samples from Gradually becomes empty, and the collection Gradually full, when collected If empty, the collection Replace recycling collection The function of the collection Replace the original collection Function; Recycling All pseudo samples (i.e., the initial pseudo sample set) and labeled samples (i.e., multiple labeled training tree crown images) obtained by semi-supervised learning are pre-loaded.

[0090] Constructing a screening set By filter set Load the pseudo samples selected by the i-th round of active learning.

[0091] Constructing a training set From the collection Randomly extract pseudo samples and filter sets Merge to form a pseudo sample set for round i learning. Set the amount of data loaded in the round training set to be fixed to A. The pseudo samples in have been used in the previous k D In the next round of active learning, the corresponding pseudo samples are deleted with increasing probability according to the image information and put into the recovery collection or when filtering the set The amount of data in the training set exceeds The amount of data is fixed. When the number of pseudo samples is greater than 1, the corresponding pseudo samples are deleted in ascending order according to the image information and put into the recovery collection. Afterwards, collect Supplement the corresponding number of pseudo samples.

[0092] The process of the i-th round of interactive learning (i.e. collaborative learning) is as follows:

[0093] (1) From the collection Randomly extract pseudo samples from the previous round of learning generated screening set Merge to form the data set for this round of learning

[0094] (2) Dataset They act on the new Teacher network and the new Student network respectively. Among them, the new Teacher network performs weak data enhancement processing (including random color transformation) on the input tree crown image and is responsible for generating new pseudo samples. The Student network performs strong data enhancement processing (including random cropping, Mixup and CutMix) on the same input tree crown image, and continuously improves its own learning ability through the joint optimization of the loss function of the Student network and the loss function of the Teacher network, that is, improves its own learning ability through the optimized loss function of the Student network. The optimized loss function is:

[0095]

[0096] in, and They represent the losses of supervised learning and unsupervised learning respectively, and γ is a hyperparameter for the trade-off between them.

[0097] (3) Obtain a pseudo sample set with high information abundance from the initial pseudo samples generated by the new Teacher network through a pseudo sample screening strategy At the same time, put the remaining low-abundance pseudo samples into The confidence threshold ρ is used to filter out repeated and uncertain pseudo samples and also put them into

[0098] (4) At the end of each round of collaborative training, the Student network is used to update the Teacher network parameters through the exponential moving average algorithm (EMA).

[0099] It should be understood that the deep collaborative learning method Deep-CLTCP (Deep Collaborative Learning for Tree Crown Detection) can improve the recognition accuracy of multiple types of tree crowns under small sample constraints. This method continuously improves the model capability by using a small-scale pseudo sample set through multi-frequency dual-model interaction, designs a new pseudo sample screening index to improve the quality of the pseudo sample set, and constructs a base model to more fully explore the richer and more detailed features of the crown. Experiments conducted using a remote sensing image set for UAV forest monitoring show that the performance of Deep-CLTCP is 3% to 7% higher than that of existing popular semi-supervised target detection methods in small sample conditions, and can also achieve 100% supervised performance using 6%-8% fewer labeled samples than these methods.

[0100] In the embodiment of the present invention, each round of learning utilizes pseudo samples with high information power of the current image and pseudo samples that are difficult for the model to distinguish in previous rounds of learning, and those pseudo samples with low information power or that are difficult for the current model to distinguish are saved separately, and reserved for the model with improved capabilities in the later stage of training (that is, after multiple rounds of learning, the current model's capabilities have been improved and it is relatively more conducive to identifying these pseudo samples).

[0101] Preferably, the S504 includes:

[0102] S5041, screening the new pseudo sample set based on the average complexity index, the average sufficiency index and the average diversity index, specifically:

[0103] The average complexity index is measured by the probability distribution entropy of the prediction results of the Teacher network model. The average complexity index is:

[0104]

[0105] in, is the average complexity, is the number of tree crown prediction bounding boxes after filtering in the i-th tree crown image, N c is the number of tree crown categories, p(c k ; b j ) is the predicted probability of the kth crown category of the jth crown prediction bounding box by the Teacher network, c k is the predicted tree crown category of the kth tree crown bounding box, b j Predict the bounding box for the jth tree crown;

[0106] The average sufficiency index is measured by the target feature layout mode of the Teacher network model, and the average sufficiency index is:

[0107]

[0108] in, is the average sufficiency, conf(b j ) and conf(c j ) are the highest confidence scores in the j-th crown bounding box predicted by the Teacher network and the highest confidence scores of its corresponding crown category; The larger it is, the more certain the tree crowns identified by the Teacher network are, which means that there is more information of interest to the model in the image.

[0109] The average diversity index is measured by the category abundance and consistency of the prediction results of the Teacher network model. The average diversity index is:

[0110]

[0111] in, is the average diversity, c j is the predicted crown category of the jth crown bounding box, |.| is the number of crown category occurrences, is the consistency of the tree crown category on the i-th tree crown image, is the number of tree crown categories predicted by the i-th tree crown image, p(c j ) is the predicted probability of the predicted crown category of the j-th crown bounding box;

[0112] S5042, performing normalization fusion processing on the average complexity index, the average sufficiency index and the average diversity index, determining the sampling weights of the average complexity index, the average sufficiency index and the average diversity index according to the normalized index values, and screening out optimized pseudo samples according to the sampling weights.

[0113] It should be understood that complexity is measured based on the entropy of the probability distribution predicted by the model; the higher the entropy, the higher the uncertainty of the network model in the prediction result, indicating that the example is more difficult; when the total entropy is the same, the fewer the number of predicted bounding boxes, the higher the average complexity. Richer information means that more visual concepts appear in the image, so the model can learn more target feature layout patterns, so the average sufficiency is used to represent it. Average diversity is used to measure the abundance of each category in the image and its consistency.

[0114] Preferably, the S5042 includes:

[0115] By average complexity The maximum and minimum values, average sufficiency The maximum and minimum values ​​and the average diversity The maximum and minimum values ​​of are normalized and calculated to obtain the normalized index value;

[0116] The normalized index value is fused using the Lp norm. When the norm p>1, the sampling weight is determined according to the normalized index value corresponding to the average complexity index, the normalized index value corresponding to the average sufficiency index, and the normalized index value corresponding to the average diversity index, and the optimized pseudo-sample is screened out according to the sampling weight.

[0117] Specifically, (1) the values ​​of each indicator are normalized to prevent the difference in value range from causing the impact of individual indicators to be too large or too small.

[0118] Take the average complexity The maximum and minimum values ​​of The maximum and minimum values ​​of The maximum and minimum values ​​of Average sufficiency and average diversity Calculation is performed to obtain a normalized value corresponding to the average complexity index, a normalized value corresponding to the average sufficiency index, and a normalized index value corresponding to the average diversity index. The normalization formula is:

[0119]

[0120] Among them, m∈{average complexity, average sufficiency, average diversity} represents the indicator type, that is, the average complexity indicator or the average sufficiency indicator, and Represent the maximum and minimum values ​​respectively, that is, the average complexity The maximum and minimum values ​​of The maximum and minimum or average diversity The maximum and minimum values ​​of represents the average complexity index, the average sufficiency index or the average diversity index;

[0121] (2) Use the Lp norm to fuse the above normalized index values ​​to represent the image information power:

[0122] s info =Sigmod(||s|| p ),

[0123]

[0124] Among them, s info is the image information force, ||s|| p The normalized values ​​of each indicator are converted into a single scalar. When the norm p>1, the indicator with a higher value will obtain more sampling weight; the Sigmod function can filter out images with larger fusion values ​​as much as possible, and can automatically adjust the screening threshold according to the differences between different data sets.

[0125] like Figure 5 As shown, an embodiment of the present invention provides a tree crown classification device based on collaborative learning, including:

[0126] A crown image set module, used to construct a training crown image set containing crown images, wherein the training crown image set includes a plurality of labeled training crown images and a plurality of unlabeled training crown images;

[0127] The initial network building module is used to build the initial Teacher network and the initial Student network with the same configuration based on the FT-Transformer architecture;

[0128] A pre-training module, used for pre-training the initial Teacher network and the initial Student network by using a plurality of the labeled training tree crown images to obtain a Teacher network and a Student network during the pre-training phase;

[0129] An update training module is used to input a plurality of the unlabeled training tree crown images into the Teacher network in the semi-supervised learning stage to generate an initial pseudo sample set, update the parameters of the Student network through the initial pseudo sample set to obtain a new Student network, and synchronously update the parameters of the new Student network to the Teacher network to obtain a new Teacher network;

[0130] An optimization module is used to construct a collaborative training data set from a plurality of the labeled training tree crown images and the initial pseudo sample set during the collaborative learning stage, perform pseudo sample screening on the collaborative training data set through the new Teacher network to generate an optimized pseudo sample set, perform optimization training on the collaborative training data set through the new Student network to obtain an optimized Student network, update the parameters of the optimized Student network to the new Teacher network, and perform training in a loop until a preset number of collaborative training times is reached to obtain an optimal Teacher network;

[0131] The classification module is used to import the tree crown image to be classified into the optimal Teacher network, and output the tree crown type result through the optimal Teacher network.

[0132] A tree crown classification device based on collaborative learning is provided in an embodiment of the present invention, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the tree crown classification method based on collaborative learning is implemented.

[0133] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the tree crown classification method based on collaborative learning as described above is implemented.

[0134] The settings of various parameters of the present invention may be:

[0135] The number of pre-training iterations is 1000; the number of iterations of semi-supervised learning does not exceed 1000; the number of newly added unlabeled images in each round of collaborative learning is k=512; the training steps of the Student network in each round of collaborative learning does not exceed 1000 times, its batch-size=64, the optimizer used is SGD, and the learning rate increases linearly from 0.001 to 0.01 in the first 600 iterations.

[0136] The amount of data loaded in the round-robin training set is A = 1024; the range of times the pseudo samples are used for training is k D = [3, 5], the deletion threshold of the filter set We used γ = 4 as the unsupervised loss for all experiments, β = 0.9996 as the EMA weight, ρ = 0.75 to filter low-quality pseudo labels, and set the number of iterations in Algorithm 1 to 4; when s info ≥0.5, pseudo samples are selected as input data of the Student network.

[0137] The following experimental data illustrate the operability and advantages of the tree crown classification method and device of the present invention. The experimental process is as follows:

[0138] The method of the present invention is evaluated using a drone forest monitoring dataset (hereinafter referred to as the dataset) in a Gaofeng forest farm in 2023. The dataset has a total of 11,162 drone remote sensing image samples of 512×512 pixels, and each sample is marked with each crown frame, that is, the corresponding crown type. Among them, the labeled frames of eucalyptus, fir, and pine account for 58.18%, 28.43%, and 13.49%, respectively. Each model performance test includes 3 tests, and each test uses 80% of the dataset as a training set and 20% as a test set. The performance mean after 3 tests is the final result of the test. Since it is necessary to observe the impact of a large amount of unlabeled data on the semi-supervised learning model, the training set is further divided into a labeled set and an unlabeled set. 2.5%, 5%, 10%, 20%, and 40% of the training set are used as labeled set data for experiments, and the remaining examples are regarded as unlabeled data. The test set is used for evaluation, and mAP (global average accuracy) is used as the performance indicator for model evaluation. All experiments use the PyTorch library and the MMDectction library.

[0139] The experimental results are:

[0140] The Deep-CLTCP model of the present invention is compared with a group of state-of-the-art supervised models and semi-supervised models. Among them, the FF-RCNN model and the YOLOv3 model are the most commonly used supervised models for tree crown recognition; the U-Teacher model, the H-Teacher model, the I-Teaching model, and the A-Teacher model are the most advanced semi-supervised target detection models, all of which use Faster-RCNN as the base model. Table 1 is a comparison table of the comprehensive mAP of the Deep-CLTCP model and other models, and the comparison results are shown in Table 1.

[0141] Table 1:

[0142] 2.5％ 5％ 10％ 20％ 40％ FF-RCNN 6.26 8.41 14.14 22.68 34.55 YOLOv3 5.48 7.28 13.21 20.12 30.16 U-Teacher 10.91 15.25 23.12 37.25 51.15 I-teaching 11.66 16.73 25.67 38.16 48.47 H-Teacher 12.75 17.38 26.37 36.32 50.61 A-Teacher 12.38 18.43 28.45 40.15 53.74 Deep-CLTCP 13.46 22.57 31.28 45.83 58.12

[0143] As shown in Table 1, with the increase of data volume, the performance of each semi-supervised model has been significantly improved, among which the model Deep-CLTCP of the present invention has the highest mAP in each stage, which is about 1.1%, 4.1%, 2.8%, 5.7% and 4.4% higher than the second-ranked model. At the same time, the increase in the amount of labeled sample data also gradually increases the growth rate of the performance of each model, and the growth rate of the model of the present invention is the largest. For example, when the labeled data increases from 20% to 40%, the growth rate of the model of the present invention is 13.7%, while the second-ranked model H-Teacher is 13.6%. This shows that the model of the present invention can more fully extract effective crown features from unlabeled data by utilizing labeled data. In addition, the performance of the two supervised learning models is significantly lower than that of the semi-supervised learning model at different stages of labeled data volume, but because FF-RCNN is a two-stage target recognition model, the relative accuracy is higher than that of YOLOv3.

[0144] The recognition accuracy of different models for different types of crowns when using 5% and 40% labeled data is given. Table 2 is a comparison table of mAP for different types of crowns when using 5% data set, and Table 3 is a comparison table of recognition performance of different types of crowns when using 40% data set. As shown in Tables 2 and 3, each model has the highest recognition success rate for eucalyptus, followed by fir. Among them, the recognition accuracy of the model of the present invention is the highest in each type of crown. When using 5% data set, Deep-CLTCP's recognition accuracy in three types of tree species is about 3.8%, 3.8% and 3.5% higher than the second-ranked model (A-Teacher); when using 40% data set, Deep-CLTCP's recognition accuracy in three types of tree species is about 3.8%, 5.8% and 5.9% higher than the second-ranked model (A-Teacher). This shows that the model of the present invention is better than other models in grasping the characteristics of easy-to-distinguish and difficult-to-distinguish tree species. Under various conditions of labeled data volume, FF-RCNN has a slightly higher recognition accuracy for various tree crowns than YOLOv3. The model developed by the present invention also introduces subnetworks such as FPN, which indirectly reflects that the base model developed by the present invention is more suitable for tree crown extraction.

[0145] Table 2:

[0146] eucalyptus cedar pine FF-RCNN 8.96 8.31 7.42 YOLOv3 7.88 6.83 5.72 U-Teacher 16.91 15.15 14.87 I-teaching 17.88 16.27 15.42 H-Teacher 18.47 17.63 16.81 A-Teacher 19.68 18.73 17.25

[0147] Table 3:

[0148] eucalyptus cedar pine FF-RCNN 36.77 34.13 32.79 YOLOv3 31.56 30.13 29.37 U-Teacher 53.56 51.46 49.17 I-teaching 49.34 48.77 46.81 H-Teacher 52.56 50.37 49.87 A-Teacher 56.34 51.45 50.86 Deep-CLTCP 60.18 57.23 56.74

[0149] Comparison of sample size under full supervision performance. In practical applications, attention is also paid to the minimum amount of labeled data required to achieve full supervision performance. Table 4 shows the full supervision performance of each model for different types of tree crown recognition and the corresponding proportion of labeled data. As shown in Table 4, the highest mAP obtained by using FPN-Faster-RCNN supervised learning with 100% full training set is 85.87%, and that of YOLOv3 is 83.25%; at this time, the present invention uses mAP equal to 83% as the full supervision performance, and observes the proportion of labeled data used when the performance of each semi-supervised learning model reaches this value. Analysis shows that the optimal accuracy of all semi-supervised learning models exceeds that of supervised learning, which shows that semi-supervised learning can force the model to have a deeper grasp of the spatial distribution of tree crowns and their morphological characteristics, and can extract more effective information from difficult samples. At the same time, the Deep-CLTCP model proposed in the present invention can achieve the performance of fully supervised learning using only 60% labeled data, which is 6% to 8% less than other semi-supervised models; moreover, the Deep-CLTCP of the present invention also exceeds other semi-supervised learning models in the recognition accuracy of the three types of tree crowns, which shows that the model of the present invention can mine more tree crown features from unlabeled data with fewer samples.

[0150] Table 4:

[0151]

[0152] The above-mentioned tree crown classification device and storage medium based on collaborative learning can refer to the implementation content and beneficial effects of the tree crown classification method based on collaborative learning specifically described above, which will not be repeated here.

[0153] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0154] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0156] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0157] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A tree crown classification method based on collaborative learning, characterized in that: The steps include: S1. constructing a training tree crown image set containing tree crown images, wherein the training tree crown image set includes a plurality of labeled training tree crown images and a plurality of unlabeled training tree crown images; S2. Build an initial Teacher network and an initial Student network with the same configuration based on the FT-Transformer architecture; S3, in the pre-training stage, pre-training the initial Teacher network and the initial Student network through a plurality of the labeled training tree crown images to obtain a Teacher network and a Student network; S4, in the semi-supervised learning stage, a plurality of the unlabeled training tree crown images are input into the Teacher network to generate an initial pseudo sample set, the parameters of the Student network are updated by the initial pseudo sample set to obtain a new Student network, and the parameters of the new Student network are synchronously updated to the Teacher network to obtain a new Teacher network; S5. In the collaborative learning stage, a collaborative training data set is constructed from a plurality of the labeled training tree crown images and the initial pseudo sample set, pseudo samples are screened for the collaborative training data set through the new Teacher network to generate an optimized pseudo sample set, the collaborative training data set is optimized and trained through the new Student network to obtain an optimized Student network, and the parameters of the optimized Student network are updated to the new Teacher network; S6, repeat S5 until the preset number of collaborative training times is reached to obtain the optimal Teacher network; S7, importing the tree crown image to be classified into the optimal Teacher network, and outputting the tree crown type result through the optimal Teacher network.

2. The tree crown classification method based on collaborative learning according to claim 1, characterized in that: After the step of obtaining the new Student network, S4 further includes: The parameters of the Student network are optimized by using the multiple labeled training tree crown images and the initial pseudo sample set, specifically: The optimization loss function of the Student network is defined as: in, and They represent the supervised learning loss of the Student network and the unsupervised learning loss of the Teacher network, respectively, and γ is and Hyperparameters that trade off between Supervised Learning Loss Function It is composed of classification losses and bounding box regressions of a plurality of the labeled training tree crown images, wherein the classification losses and bounding box regressions of the plurality of the labeled training tree crown images are obtained through labels of the labeled training tree crown images; Unsupervised learning loss It is composed of the classification loss of the initial pseudo sample set, and the classification loss of the initial pseudo sample set is obtained by the pseudo labels of the initial pseudo sample set.

3. The tree crown classification method based on collaborative learning according to claim 1, characterized in that: The S5 comprises: S501, constructing a collaborative training data set from the plurality of labeled training tree crown images and the initial pseudo sample set, wherein the collaborative training data set includes a recovered set, a screened set, and a rotation training set; S502, extracting the collected collaborative recovery data according to preset parameters, obtaining the collaborative screening data of the previous round of collaborative training in the screening set, and merging the collaborative recovery data and the collaborative screening data into a round training set; S503, performing weak data enhancement processing on the training set through the new Teacher network, and performing pseudo sample screening on the enhanced training set to generate a new pseudo sample set, performing strong data enhancement processing on the training set using the new Student network, and learning the enhanced training set to obtain an optimized Student network; S504, screening the new pseudo sample set according to the pseudo sample screening strategy to obtain optimized pseudo samples; S505: Update the parameters of the optimized Student network to the new Teacher network through an exponential moving average algorithm (EMA).

4. The tree crown classification method based on collaborative learning according to claim 3 is characterized in that: The S501 includes: Constructing a first collection from a plurality of the labeled training tree crown images and the initial pseudo sample set; When the collaborative screening data in the training set has been learned by the new Student network, the corresponding sample data is deleted from the training set according to the image information of the collaborative screening data and stored in the next collection; When the number of the collaborative screening data in the round-robin set exceeds the preset round-robin number, the corresponding sample data is deleted from the round-robin set according to the image information of the collaborative screening data and stored in the next collection; The new pseudo samples screened out according to the pseudo sample screening strategy are stored in the next collection; The optimized pseudo samples are used as collaborative screening data for the current round of collaborative training and stored in the screening set.

5. The tree crown classification method based on collaborative learning according to claim 3, characterized in that: The S504 includes: S5041, screening the new pseudo sample set based on the average complexity index, the average sufficiency index and the average diversity index, specifically: The average complexity index is measured by the probability distribution entropy of the prediction results of the Teacher network model. The average complexity index is: in, is the average complexity, is the number of tree crown prediction bounding boxes after filtering in the i-th tree crown image, N c is the number of tree crown categories, p(c k ; b j ) is the predicted probability of the k-th tree crown category by the Teacher network; The average sufficiency index is measured by the target feature layout mode of the Teacher network model, and the average sufficiency index is: in, is the average sufficiency, conf(b j ) and conf(c j ) are the highest confidence scores in the j-th crown bounding box predicted by the Teacher network and the highest confidence scores of its corresponding crown categories, c j is the predicted crown category of the jth crown bounding box; The average diversity index is measured by the category abundance and consistency of the prediction results of the Teacher network model. The average diversity index is: in, is the average diversity, c j is the predicted crown category of the jth crown bounding box, |.| is the number of crown category occurrences, is the consistency of the tree crown category on the i-th tree crown image, is the number of tree crown categories predicted for the i-th tree crown image; S5042, performing normalization fusion processing on the average complexity index, the average sufficiency index and the average diversity index, determining the sampling weights of the average complexity index, the average sufficiency index and the average diversity index according to the normalized index values, and screening out optimized pseudo samples according to the sampling weights.

6. The tree crown classification method based on collaborative learning according to claim 5, characterized in that: The S5042 includes: By average complexity The maximum and minimum values, average sufficiency The maximum and minimum values ​​and the average diversity The maximum and minimum values ​​of are normalized and calculated to obtain the normalized index value; The normalized index value is fused using the Lp norm. When the norm p>1, the sampling weight is determined according to the normalized index value corresponding to the average complexity index, the normalized index value corresponding to the average sufficiency index, and the normalized index value corresponding to the average diversity index, and the optimized pseudo-sample is screened out according to the sampling weight.

7. A tree crown classification device based on collaborative learning, characterized in that: include: A crown image set module, used to construct a training crown image set containing crown images, wherein the training crown image set includes a plurality of labeled training crown images and a plurality of unlabeled training crown images; The initial network building module is used to build the initial Teacher network and the initial Student network with the same configuration based on the FT-Transformer architecture; A pre-training module, used for pre-training the initial Teacher network and the initial Student network by using a plurality of the labeled training tree crown images to obtain a Teacher network and a Student network during the pre-training phase; An update training module is used to input a plurality of the unlabeled training tree crown images into the Teacher network in the semi-supervised learning stage to generate an initial pseudo sample set, update the parameters of the Student network through the initial pseudo sample set to obtain a new Student network, and synchronously update the parameters of the new Student network to the Teacher network to obtain a new Teacher network; An optimization module is used to construct a collaborative training data set from a plurality of the labeled training tree crown images and the initial pseudo sample set during the collaborative learning stage, perform pseudo sample screening on the collaborative training data set through the new Teacher network to generate an optimized pseudo sample set, perform optimization training on the collaborative training data set through the new Student network to obtain an optimized Student network, update the parameters of the optimized Student network to the new Teacher network, and perform training in a loop until a preset number of collaborative training times is reached to obtain an optimal Teacher network; The classification module is used to import the tree crown image to be classified into the optimal Teacher network, and output the tree crown type result through the optimal Teacher network.

8. The tree crown classification device based on collaborative learning according to claim 1, characterized in that: The updating training module further includes: The parameters of the Student network are optimized by using the multiple labeled training tree crown images and the initial pseudo sample set, specifically: The optimization loss function of the Student network is defined as: in, and They represent the supervised learning loss of the Student network and the unsupervised learning loss of the Teacher network, respectively, and γ is and Hyperparameters that trade off between Supervised Learning Loss Function It is composed of classification losses and bounding box regressions of a plurality of the labeled training tree crown images, wherein the classification losses and bounding box regressions of the plurality of the labeled training tree crown images are obtained through labels of the labeled training tree crown images; Unsupervised learning loss It is composed of the classification loss of the initial pseudo sample set, and the classification loss of the initial pseudo sample set is obtained by the pseudo labels of the initial pseudo sample set.

9. A tree crown classification device based on collaborative learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the tree crown classification method based on collaborative learning as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the tree crown classification method based on collaborative learning as described in any one of claims 1 to 6 is implemented.

Images