Forest scene segmentation model training method, segmentation method, medium and device

Abstract

The application discloses a kind of forest scene segmentation model training method, segmentation method, medium and equipment, belong to deep learning technical field.The first feature extraction module is extracted to obtain the first feature representation by using feature extraction;First feature representation is input into classification head;First feature representation is input into first mapping module to obtain first mapping feature and act sampling strategy, obtain sampling mapping feature;Sampling mapping feature is compared with the feature in feature vector library, and the similarity of comparison is calculated;First mapping feature is input into prediction module to obtain predicted feature representation;After input image is enhanced, it is input into second feature extraction module to obtain second feature representation by feature extraction;Second feature representation is input into second mapping module to obtain second mapping feature;The similarity of predicted feature representation and second mapping feature is calculated, and the similarity of predicted feature representation and second mapping feature is maintained.The precision and generalization ability of the forest scene segmentation model are improved.

Description

Technical Field

[0001] This belongs to the field of deep learning technology, specifically involving a training method, segmentation method, medium, and device for a forest scene segmentation model. Background Technology

[0002] Forest scene segmentation plays a crucial role in forest-related tasks such as fire detection and wildlife detection. Inaccurate forest delineation significantly reduces the accuracy of subsequent detection tasks, leading to missed or false detections and increasing manual workload. Traditional machine learning methods, such as gray-scale thresholding, support vector machines, or K-class clustering, are primarily used for forest scene segmentation, but these only achieve accurate segmentation of two classes: the forest class and the other class. With the development of deep learning, computer vision has made significant strides. In particular, the widespread application of convolutional networks has greatly improved object detection and semantic segmentation. Therefore, using deep learning techniques to solve forest scene segmentation has attracted widespread attention from both academia and industry.

[0003] However, forest scene segmentation differs from other scene segmentation. In forest scenes, most objects are forests themselves; categories like lakes and roads are rare or have small areas in the image, leading to inaccurate segmentation results for categories other than forests. Furthermore, forest scenes are often shrouded in fog, resulting in long acquisition distances and blurry images during actual testing, further increasing the difficulty of segmentation. Therefore, many current forest scene segmentation methods utilize large-capacity models to improve segmentation accuracy, but this does not allow for real-time segmentation and incurs high model deployment costs. Summary of the Invention

[0004] Technical problem: This invention provides a training method that can improve the accuracy and generalization ability of a forest scene segmentation model, a method for performing forest scene segmentation using the model trained by this method, and corresponding storage media and electronic devices.

[0005] Technical solution: In a first aspect, the present invention provides a method for training a forest scene segmentation model, comprising:

[0006] The input image is input into the first feature extraction module for feature extraction to obtain the first feature representation;

[0007] Input the first feature representation into the classification head to obtain the classification mask and classification confidence.

[0008] The first feature representation is input into the first mapping module to obtain the first mapped feature;

[0009] The first mapping feature is applied to the sampling strategy to obtain the sampled mapping feature; wherein, the method of applying the first mapping feature to the sampling strategy to obtain the sampled mapping feature is to compare the classification mask and the reference classification mask to filter out the features corresponding to pixels with incorrect classification results and correct classification results but classification confidence below the threshold.

[0010] The features in the feature vector library are sorted by center distance, and different weights are assigned to each feature in the feature vector library. The sampled mapping features are compared with the features of each category in the feature vector library, and the comparison similarity is calculated.

[0011] The first mapping feature is input into the prediction module to obtain the predicted feature representation;

[0012] After feature enhancement, the input image is input into the second feature extraction module for feature extraction to obtain the second feature representation.

[0013] The second feature representation is input into the second mapping module to obtain the second mapping feature;

[0014] Calculate the similarity between the predicted feature representation and the second mapped feature, and maintain the similarity between the predicted feature representation and the second mapped feature.

[0015] Furthermore, the features in the feature vector library are sorted by center distance, and different weights are assigned to each feature in the vector library as follows:

[0016] In the feature vector library, the feature centers of each category are obtained by summing and averaging the features of each category.

[0017] For each feature in each category, sort them from closest to furthest based on the normalized L2 distance from the feature center of the category;

[0018] The weight of each feature is obtained by subtracting the maximum distance from the center in each category from the distance of each feature, exponentializing the result, and dividing by the average distance of the same category.

[0019] Furthermore, the sampling mapping features are compared with the features of each category in the feature vector library, and the method for calculating the comparison similarity is as follows:

[0020]

[0021] Where e represents a certain sampling mapping feature, e + Features that belong to the same category as this feature are called positive samples; e - Features that belong to a different category from the given feature are called negative samples. The feature weights represent the positive samples. Represents the feature weights of negative samples; It is the set of positive sample features. The number of positive sample features is represented by z; the set of sampled mapping features is represented by N; Contr is the comprehensive similarity value between the sampled features and various features in the feature library; and τ is the temperature coefficient.

[0022] Further, a method for calculating the similarity between the predicted feature representation and the second mapped feature:

[0023]

[0024] Where z1 is the predicted feature representation, y2 is the second mapping feature, ||·||2 represents the Euclidean distance, and Cons represents the cosine similarity between the two.

[0025] Furthermore, it also includes using a feature vector library to store feature representations of various categories in the forest scene, filtering the first mapping features through a filter, and putting the filtered features into the feature vector library for real-time updates.

[0026] Furthermore, the method for updating the feature vector library in real time based on feature confidence is as follows:

[0027] Extract the correctly classified features and those with a confidence level greater than the threshold from the first mapping features, add them to the feature vector library, and update the feature vector library.

[0028] Secondly, this invention improves a forest scene segmentation method, which uses a forest scene segmentation model trained by the training method proposed in this invention to segment a forest scene, including:

[0029] Acquire forest scene images;

[0030] The forest scene image is input into the trained forest scene segmentation model, and the segmentation result is output.

[0031] Thirdly, the present invention provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, cause the processor to execute the training method of the forest scene segmentation model; or the provided forest scene segmentation method.

[0032] Fourthly, the present invention provides an electronic device including a computer-readable storage medium and a processor capable of executing computer program instructions stored in the computer-readable storage medium.

[0033] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0034] (1) The training method of the forest scene segmentation model of the present invention has a main branch and an auxiliary branch during training. During training, the input image of the main branch is a sample without image enhancement, and the input image of the auxiliary branch is a sample with image enhancement. The cosine similarity between the output features of the two is calculated during training. The consistency of the features of the two is required, which can increase the noise resistance of the model. Thus, the trained model can obtain consistent segmentation results in different ambient lighting conditions when applied in engineering.

[0035] Furthermore, the training method for the forest scene segmentation model of this invention also employs contrastive learning and a feature vector library during training. Using a sampling strategy, a subset of features obtained from the main branches are selected and compared with similar and dissimilar features in the feature vector library. This ensures consistent feature representations within the same category, while different categories exhibit distinct feature representations. Simultaneously, the feature vector library is updated, thereby improving the segmentation accuracy for each category. At the same time, the sampling strategy used in feature selection significantly reduces unnecessary computational costs during model training, ensuring that model training remains at a low computational cost level.

[0036] (2) The forest scene segmentation method of the present invention utilizes the forest scene segmentation model trained by the forest scene segmentation model training method proposed in this invention, thereby enabling more accurate segmentation of forest scenes. In addition, since the forest scene segmentation model is only the first feature extraction module and the classification head, the computational cost of the model is significantly reduced compared to the training process when it is deployed, and the segmentation accuracy remains consistent. Therefore, the application and deployment of this method do not require additional costs compared to traditional methods.

[0037] (3) The forest scene segmentation electronic device proposed in this invention can segment the forest scene more accurately. When deployed in a specific application scenario, it can accurately segment the forest scene, thereby better assisting in tasks such as forest fire detection and greatly reducing the amount of manual work. Attached Figure Description

[0038] Figure 1 This is a network structure framework diagram of the training method for the forest scene segmentation model in an embodiment of the present invention;

[0039] Figure 2 This is a flowchart illustrating the training method of the forest scene segmentation model in an embodiment of the present invention;

[0040] Figure 3 This is a flowchart of a forest scene segmentation method in an embodiment of the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the embodiments and accompanying drawings. It should be noted that the terms "first," "second," etc., used in the embodiments are for ease of description only and should not be construed as limiting the quantity, properties, etc.

[0042] First, the structure of the forest scene segmentation model involved in the embodiments of the present invention will be described. In the embodiments of the present invention, the forest scene segmentation model is an existing convolutional neural network model, including a feature extraction module and a classification head. The forest scene image is input into the trained forest scene segmentation model. First, the feature extraction module performs feature extraction, and then the classification head performs classification prediction. Currently, the conventional training method is to directly train the forest scene segmentation model using a dataset to obtain a trained forest scene segmentation model. However, the model trained in this way has insufficient accuracy and generalization ability. One of the main objectives of the present invention is to provide a new training method for training the forest scene segmentation model to improve accuracy and generalization ability. To this end, in the embodiments of the present invention, a network structure for training is constructed, such as... Figure 1 As shown, the forest scene segmentation model is a dual-path model during training, consisting of a main branch for predicting segmentation results, an auxiliary branch for enhancing the model's generalization ability, and a feature vector library for enhancing the model's discrimination ability. The main branch includes the first feature extraction module, the first mapping module, and the prediction module shown in the figure; the auxiliary branch includes the second feature extraction module and the second mapping module.

[0043] In one embodiment of the present invention, the first feature extraction module and the second feature extraction module are identical, employing the DeepLabV3Plus network structure and using MobileNetV2 as the pre-feature extraction module. The first mapping module and the second mapping module are also identical; together with the prediction module, all three are fully convolutional modules. It should be noted that in other embodiments, those skilled in the art can replace the DeepLabV3Plus network structure and the MobileNetV2 feature extraction module with other existing network structures and other pre-feature extraction modules.

[0044] Figure 2 A flowchart illustrating the training method of the forest scene segmentation model in an embodiment of the present invention is shown. Combined with... Figure 1 As shown, in this embodiment, the training method includes steps S100 to S180:

[0045] Step S100: Input the input image into the first feature extraction module for feature extraction to obtain a first feature representation. In one embodiment of the present invention, the first feature extraction module adopts the network structure of DeepLabV3Plus and uses MobileNetV2 as a pre-feature extraction module, as expressed by the following formula:

[0046] (F low ,F high )=f AC (IF)

[0047] Among them, F low ,F high This is the feature representation of the input image IF obtained through the MobileNetV2 network. The input image has dimensions of 3×H×W, with 3 color image channels. H and W are the height and width of the input image, respectively, and F... low ,F high These are low-dimensional and high-dimensional features, respectively. In this embodiment, after MobileNetV2 extracts features from the input image, it obtains feature representations at two scales. Then, it uses the Spatial Dilated Convolutional Pooling Pyramid (ASPP) to convert the high-dimensional features F... high The transformation is performed, and the feature is fused with the upsampled low-dimensional feature to form the total feature F. a , represented as:

[0048] F a =ASPP(F high )+Upsample(F low )

[0049] In an embodiment of the present invention, a first feature extraction module extracts feature representations of the input image. High-dimensional features are processed through dilated spatial convolution pooling pyramids, and low-dimensional features are upsampled. The two feature representations are then fused to extract the total feature F of the input image. a Furthermore, the total characteristic F a The size is

[0050] Step S110: Input the first feature representation into the classification head to obtain the classification mask and classification confidence.

[0051] Step S120: Input the first feature representation into the first mapping module to obtain the first mapped feature. In an embodiment of the present invention, the first mapping module consists of a fully convolutional network (FCN), represented as:

[0052] F proj =FCN(F a )

[0053] Among them, F aThe total features are obtained by passing the input image through MobileNetV2 and dilated spatial convolutional pooling pyramids. FCN is a fully convolutional network. proj For mapping features, the size is

[0054] Step S130: Apply the first mapping feature to the sampling strategy to obtain the sampled mapping feature. Specifically, compare the classification mask and the reference classification mask to filter out the features corresponding to pixels with incorrect classification results and pixels with correct classification results but classification confidence below the threshold.

[0055] Step S140: Features in the feature vector library are sorted by center distance, and different weights are assigned to each feature. The sampled mapped features are compared with the features of each category in the feature vector library to calculate the comparison similarity. Specifically, this includes steps S141 to S144, where steps S141 to S143 are used to sort the features in the feature vector library by center distance and assign different weights to each feature, and step S144 is used to compare the sampled mapped features with the features of each category in the feature vector library to calculate the comparison similarity.

[0056] Step S141: In the feature vector library, sum and average the features of each category to obtain the feature center of each category.

[0057]

[0058] Among them, f i Let n represent the feature of the i-th category in the feature library, and n be its number.

[0059] Step S142: For each feature in each category, sort them from closest to furthest according to the normalized L2 distance from the feature center of the category, using the following formula:

[0060]

[0061] Where l2-norm.(*) represents the normalized L2 distance. This represents the j-th feature of the i-th category. This indicates the distance of the feature from the center of the same class.

[0062] Step S143: Subtract the distance of each feature from the maximum distance from the center in each category, exponentialize the result, and divide by the mean distance of the same category to obtain the weight of each feature. The formula is as follows:

[0063]

[0064] Among them, dist i Let k represent the distance set in each category, and k represent the number of distances.

[0065] S144: The sampled mapping features are compared with the features of each category in the feature vector library to calculate the comparison similarity. In one embodiment of the present invention, the method for comparison similarity is expressed as follows:

[0066]

[0067] Where e represents F proj A certain feature obtained through the sampling strategy, e + Features in the feature vector library that belong to the same category as this feature are called positive samples; e - A feature in the feature vector library that belongs to a different category from the given feature is called a negative sample. The feature weights represent the positive samples. Represents the feature weights of negative samples. It is the set of positive sample features. z represents the number of features of positive samples, and z represents F. proj The feature set obtained through the sampling strategy is represented by N, which indicates the number of features; Contr represents the comprehensive similarity value between the sampled features and various features in the feature library; and τ represents the temperature coefficient. Using the above formula, the similarity between features of the same category is increased, while the dissimilarity between features of different categories is increased.

[0068] In a preferred embodiment of the present invention, after step S140, a feature vector library can be used to store feature representations of various categories in the forest scene. The first mapping feature is filtered through a filter, and the filtered features are placed into the feature vector library for real-time updates. Specifically, the method involves starting from the first mapping feature F... proj Features that are correctly classified and have a confidence level greater than a threshold are extracted and added to the feature vector library, which is then updated.

[0069] Step S150: Input the first mapping feature into the prediction module to obtain the predicted feature representation.

[0070] Step S160: After feature enhancement, the input image is input into the second feature extraction module for feature extraction to obtain a second feature representation. In the embodiments of the present invention, the feature extraction method of the second feature extraction module in the auxiliary branch is the same as the feature extraction method in the first feature extraction module of the main branch, specifically:

[0071]

[0072] Among them, IF aug Indicates the input image. This is the feature representation obtained by processing the input image through the MobileNetV2 network. The input image has dimensions of 3×H×W, with 3 color image channels. H and W are the height and width of the input image, respectively, and F... low ,F high These are low-dimensional and high-dimensional features, respectively. In this embodiment, after MobileNetV2 extracts features from the input image, it obtains feature representations at two scales. Then, it uses the Spatial Dilated Convolutional Pooling Pyramid (ASPP) to convert the high-dimensional features F... high The transformation is performed, and the features are fused with the upsampled low-dimensional features to form the total features. Represented as:

[0073]

[0074] General characteristics The size is

[0075] Step S170: Input the second feature representation into the second mapping module to obtain the second mapping feature.

[0076] Step S180: Calculate the similarity between the predicted feature representation and the second mapped feature, while maintaining the similarity between the predicted feature representation and the second mapped feature. Specifically, in one embodiment of the present invention, the cosine similarity between the predicted feature representation and the second mapped feature is calculated. The method for calculating cosine similarity is as follows:

[0077]

[0078] Where z1 represents the predicted feature representation, y2 represents the second mapping feature, ||·||2 represents the Euclidean distance, and Cons represents the cosine similarity between the two branches.

[0079] In an embodiment of the present invention, the features output by the auxiliary branch are the features of the input image after image enhancement, while the features output by the main branch are the features of the original input image. Maintaining the cosine similarity between the two can increase the robustness of the model and correctly predict the type under different light intensity contrast conditions.

[0080] The model trained using the training method of the forest scene segmentation model of this invention has a dual-path model during training. During the training process, the addition of a feature vector library allows the features to be classified to be compared with the features of each category in the feature vector library, narrowing the distance between features of the same category and making the distribution of different categories more dispersed, increasing the recognition range of each category, and improving the accuracy of the model. This solves the problem of segmenting difficult-to-distinguish samples in forest scenes.

[0081] Furthermore, the inclusion of auxiliary branches constrains the cosine similarity between the features of the main branch and the features of the auxiliary branches. This enhances the consistency of prediction results for the same scene under different weather conditions and at different times, improving the model's robustness and broadening its application scenarios. Moreover, during detection, the trained model removes the mapping and prediction modules of the main branch, the auxiliary branches, and the feature vector library, retaining only the first feature extraction module from the main branch. This allows for accurate segmentation of forest scenes, ensuring low computational costs and facilitating engineering deployment.

[0082] The following table shows a comparison of the accuracy of the forest scene segmentation model trained using the training method of the forest scene segmentation model in the embodiments of the present invention and the forest scene segmentation model trained using conventional methods, with the average intersection-union ratio mIoU as the evaluation index.

[0083]

[0084] In the table, FSSNet represents the results of forest scene segmentation by the model trained in the embodiments of the present invention. As can be seen from the table, the average crossover ratio (mIoU) of the forest scene segmentation model trained using the training method proposed in the present invention is relatively large, indicating that the accuracy and generalization ability of the trained forest scene segmentation model are relatively good.

[0085] Secondly, embodiments of the present invention provide a forest scene segmentation method, which segments a forest scene using a forest scene segmentation model trained by the training method proposed in this invention. The method includes:

[0086] Step S200: Acquire an image of the forest scene, for example, by taking a picture of the forest scene with a camera.

[0087] Step S210: Input the forest scene image into the trained forest scene segmentation model and output the segmentation result. Specifically, after the forest scene image is input into the trained first feature extraction module, the first image features are extracted, and then the classification head is used to obtain a classification mask. The forest scene is then segmented according to the feature mask.

[0088] As can be seen from the table above, this forest scene segmentation method can segment forest scenes more accurately.

[0089] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, cause the processor to execute the training method for the forest scene segmentation model or the forest scene segmentation method described in this specification.

[0090] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0091] Fourthly, embodiments of the present invention provide an electronic device, specifically including the aforementioned computer-readable storage medium and a processor. The processor is capable of executing computer instructions stored in the computer-readable storage medium. When the computer instructions are executed, the processor is capable of training a forest scene segmentation model or segmenting a forest scene according to the method proposed in the present invention.

[0092] The above embodiments are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and equivalent substitutions without departing from the principle of the present invention. All such improvements and equivalent substitutions to the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A training method for a forest scene segmentation model, characterized in that, include: The input image is input into the first feature extraction module for feature extraction to obtain the first feature representation; Input the first feature representation into the classification head to obtain the classification mask and classification confidence. The first feature representation is input into the first mapping module to obtain the first mapped feature; The first mapping feature is applied to the sampling strategy to obtain the sampled mapping feature; wherein, the method of applying the first mapping feature to the sampling strategy to obtain the sampled mapping feature is to compare the classification mask and the reference classification mask to filter out the features corresponding to pixels with incorrect classification results and correct classification results but classification confidence below the threshold. The features in the feature vector library are sorted by center distance, and different weights are assigned to each feature in the feature vector library. The sampled mapping features are compared with the features of each category in the feature vector library, and the comparison similarity is calculated. The first mapping feature is input into the prediction module to obtain the predicted feature representation; After feature enhancement, the input image is input into the second feature extraction module for feature extraction to obtain the second feature representation. The second feature representation is input into the second mapping module to obtain the second mapping feature; Calculate the similarity between the predicted feature representation and the second mapped feature, and maintain the similarity between the predicted feature representation and the second mapped feature.

2. The training method according to claim 1, characterized in that, The features in the feature vector library are sorted by center distance, and different weights are assigned to each feature in the vector library as follows: In the feature vector library, the feature centers of each category are obtained by summing and averaging the features of each category. For each feature in each category, sort them from closest to furthest based on the normalized L2 distance from the feature center of the category; The weight of each feature is obtained by subtracting the maximum distance from the center in each category from the distance of each feature, exponentializing the result, and dividing by the average distance of the same category.

3. The training method according to claim 2, characterized in that, The method for calculating the similarity between the sampled mapping features and the features of each category in the feature vector library is as follows: Where e represents a certain sampling mapping feature, e + Features that belong to the same category as this feature are called positive samples; e - Features that belong to a different category from the given feature are called negative samples. Represents the feature weights of positive samples. Represents the feature weights of negative samples; It is the set of positive sample features. Indicates the number of features in a positive sample; τ represents the sampled mapping feature set, N represents its number, Contr represents the comprehensive similarity value between the sampled features and various features in the feature library, and τ represents the temperature coefficient.

4. The training method according to claim 3, characterized in that, A method for calculating the similarity between the predicted feature representation and the second mapped feature: Where z1 is the predicted feature representation, y2 is the second mapping feature, ‖·‖2 represents the Euclidean distance, and Cons represents the cosine similarity between the two.

5. The training method according to any one of claims 1-4, characterized in that, It also includes using a feature vector library to store feature representations of various categories in the forest scene, filtering the first mapping features through a filter, and putting the filtered features into the feature vector library for real-time updates.

6. The training method according to claim 5, characterized in that, The method for updating the feature vector library in real time based on feature confidence is as follows: Extract the correctly classified features and those with a confidence level greater than the threshold from the first mapping features, add them to the feature vector library, and update the feature vector library.

7. A forest scene segmentation method, comprising segmenting a forest scene using a forest scene segmentation model trained by the training method of any one of claims 1-6, characterized in that, include: Acquire forest scene images; The forest scene image is input into the trained forest scene segmentation model, and the segmentation result is output.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, cause the processor to perform the training method for the forest scene segmentation model as described in any one of claims 1-6; or the forest scene segmentation method as described in claim 7.

9. An electronic device, characterized in that, The invention includes the computer-readable storage medium of claim 8, and a processor capable of executing computer program instructions stored in the computer-readable storage medium.

Images