Tree remote sensing image processing method and system based on primary sample learning

Abstract

The invention relates to the field of image recognition, and discloses a tree remote sensing image processing method and system based on primary sample learning, and the method comprises the steps: designing a general drawing segmentation model DrawSAM based on a basic model SAM, carrying out the self-adaptive fine adjustment of the SAM through employing a high-efficiency fine adjustment method DoRA, carrying out the image segmentation through combining with a Box Generator prompt generator, and carrying out the recognition of a tree remote sensing image. The method can adapt to diversified drawing types and complex structure layouts, and the detection and recognition accuracy and efficiency of characters and symbols in the drawings are improved; and through the optimized text detection algorithm DrawDETR, the error recognition rate during engineering drawing recognition is reduced, the recognition accuracy is improved, the problems of error recognition and incapability of symbol recognition of a general OCR model in the field are effectively solved, retrieval and application of drawing electronic data are remarkably improved, and industrial requirements are met.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a tree remote sensing image processing method and system based on one-time sample learning. Background Art

[0002] At present, there are two main technologies for estimating the height of tree remote sensing images. One is based on deep learning technology, which trains models to learn the feature differences between tree trunks and backgrounds, thereby achieving accurate segmentation, and then obtaining the tree height through the tree outline. The other is based on 3D point cloud technology. The 3D point cloud data of trees is obtained through methods such as laser scanning, and then the point cloud processing algorithm is used to segment the trunk structure, and then the height information is obtained through the tree outline.

[0003] However, the above methods still have some shortcomings and deficiencies, mainly including the following aspects: Limited data volume: The performance of deep learning models is highly dependent on the quantity and quality of training data. For tree extraction tasks of specific tree species or under specific environmental conditions, a large amount of labeled data may be required to train the model. Limited model generalization ability: When a deep learning model encounters a new scene that is significantly different from the training data, the performance of the model may decline. Limited computing resources: Deep learning models require long-term training and high computing resources, and are not suitable for use in resource-constrained environments. High equipment cost: If three-dimensional scanning equipment such as lidar is used to obtain tree height, this equipment is expensive and not suitable for large-scale popularization. Limitations on the complexity of multiple trees: Traditional methods are generally used to extract single trees in images. For complex images containing multiple trees, the extraction accuracy is low.

[0004] Therefore, an image processing method is needed that has high generalization ability and high prediction accuracy with less resources. Summary of the invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a tree remote sensing image processing method and system based on one-time sample learning to solve the problems of inaccurate prediction and weak generalization ability of deep learning models caused by limited resources and data volume.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a tree remote sensing image processing method based on one-shot sample learning, comprising: acquiring tree images, and performing correction to obtain a first image set;

[0009] Select any one of the first image set as a sample image, perform tree marking, and obtain a mask image; and extract features of the sample image to obtain first tree contour features;

[0010] The mask image is fused with the first tree contour feature to obtain an image matrix;

[0011] Apply global averaging and maximum pooling to the image matrix to obtain a first tree outline visual embedding value; use the remaining images in the first image set as test images to extract a second tree outline feature; and calculate by combining the first tree outline visual embedding value and the second tree outline feature to obtain a first pixel coordinate;

[0012] Input the sample image and the first pixel coordinates into the prediction model and output the first mask; if there are multiple tree segmentation tasks, determine the intersection-over-union ratio of the first mask and the test image mask, perform multiple predictions based on the determination results, and output all masks;

[0013] The tree outline features are obtained based on the output mask to calculate the true height of the tree. As a preferred solution of the tree remote sensing image processing method based on one-sample learning described in the present invention, wherein: applying global averaging and maximum pooling to the image matrix to obtain the first tree outline visual embedding value includes:

[0014] Apply global average pooling to the image matrix, average all elements in each channel, and obtain the average output; where the output of each channel is the first output;

[0015] Apply global maximum pooling to the image matrix, find the maximum value of all elements in each channel, and obtain the maximum output; where the output of each channel is the second output;

[0016] The first output is combined with the second output by weighted summation to obtain a first tree silhouette visual embedding value, wherein the global average pooling and the global maximum pooling weight coefficients are the same.

[0017] As a preferred solution of the tree remote sensing image processing method based on one-time sample learning described in the present invention, wherein: combining the first tree outline visual embedding value and the second tree outline feature to calculate and obtain the first pixel coordinate includes:

[0018] For each c-dimensional vector in the second tree outline feature, the cosine similarity is calculated; the pixel coordinates of a point with the highest cosine similarity are selected as the first pixel coordinates, which are used as the prediction model prompt input points.

[0019] As a preferred solution of the tree remote sensing image processing method based on one-time sample learning described in the present invention, wherein: the sample image and the first pixel coordinate are input into the prediction model, and the first mask is output, including: introducing an attention mechanism into the prediction model, performing attention feature extraction, and obtaining a first prediction feature;

[0020] Input the first predicted feature into the decoding layer to obtain a preliminary mask;

[0021] Perform light fine-tuning on the preliminary mask and output the first mask.

[0022] As a preferred solution of the tree remote sensing image processing method based on one-time sample learning described in the present invention, wherein: any one of the first image set is selected as a sample image, and tree annotation is performed to obtain a mask image, including:

[0023] Performing a first judgment on the selected sample image;

[0024] If it is determined to be a single tree segmentation task, the single tree in the center of the image is marked to obtain a mask image containing a single tree;

[0025] If it is determined to be a multi-tree segmentation task, all trees with clear structures in the image are marked to obtain a mask image containing multiple trees.

[0026] As a preferred solution of the tree remote sensing image processing method based on one-sample learning of the present invention, if there is a multi-tree segmentation task, the intersection-over-union ratio of the first mask and the test image mask is determined, multiple predictions are performed based on the determination results, and all masks are output, including:

[0027] If the intersection-union ratio is greater than the first threshold, it is considered as a repeated mask and is not saved; the next round of prediction is performed until the number of masks reaches the maximum limit, and the prediction is terminated; all saved masks are output.

[0028] As a preferred solution of the tree remote sensing image processing method based on one-time sample learning described in the present invention, wherein: tree contour features are obtained based on the output mask to calculate the true height of the tree, including: extracting all tree contour features in all output masks through an edge detection method; selecting the largest contour among all tree contour features to obtain pixel height scale information; and obtaining the true height of the tree based on triangulated imaging calculation.

[0029] In a second aspect, the present invention provides a tree remote sensing image processing system based on one-shot sample learning, comprising: an acquisition module, used to acquire tree images, perform correction to obtain a first image set; a labeling module, used to select any one of the first image sets as a sample image, perform tree labeling to obtain a mask image; and extract sample image features to obtain first tree outline features;

[0030] A first fusion module is used to fuse the mask image with the first tree outline feature to obtain an image matrix; a pooling module is used to apply global average and maximum pooling to the image matrix to obtain a first tree outline visual embedding value;

[0031] An extraction module, used for taking the remaining images in the first image set as test images, and extracting the second tree outline features;

[0032] A first calculation module, used to calculate by combining the first tree outline visual embedding value and the second tree outline feature to obtain a first pixel coordinate;

[0033] A prediction module, configured to input the sample image and the first pixel coordinates into a prediction model and output a first mask;

[0034] A judgment module, used for judging the intersection-and-union ratio of the first mask and the test image mask if there are multiple tree segmentation tasks, making multiple predictions based on the judgment results, and outputting all masks;

[0035] The second calculation module is used to obtain tree outline features based on the output mask to calculate the actual height of the tree.

[0036] In a third aspect, the present invention provides an electronic device, comprising:

[0037] Memory and processor;

[0038] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the tree remote sensing image processing method based on one-time sample learning are implemented.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the tree remote sensing image processing method based on one-time sample learning.

[0040] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes a tree remote sensing image processing model based on one-time sample learning, which is used to obtain the contour features of a single tree or multiple trees, and to link the contour features with distance parameters, providing a method with higher performance in the field of tree growth analysis of single trees and multiple trees; it solves the problem that traditional methods require a lot of training, high labeling cost and high time cost; at the same time, through coordinate mapping, a system is realized that accurately calculates the real tree height through images; the parameters can also be lightly fine-tuned to adapt to task scenarios in complex tree growth environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0042] Figure 1 A textual schematic diagram of the overall process of a tree remote sensing image processing method based on one-time sample learning according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the overall flow structure of a tree remote sensing image processing method based on one-time sample learning according to an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the structure of the attention feature layer in the tree remote sensing image processing method based on one-time sample learning according to an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of the structure of a decoding layer in a tree remote sensing image processing method based on one-time sample learning according to an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of visualization results of single tree segmentation in a tree remote sensing image processing method based on one-sample learning according to an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of visualization results of multi-tree segmentation in a tree remote sensing image processing method based on one-sample learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0049] Example 1

[0050] Reference Figure 1 , which is an embodiment of the present invention, provides a tree remote sensing image processing method based on one-time sample learning, comprising:

[0051] S100: Acquire tree images and perform correction to obtain a first image set;

[0052] S200: Select any one of the first image set as a sample image, perform tree marking, and obtain a mask image; and extract features of the sample image to obtain first tree outline features;

[0053] S300: fusing the mask image with the first tree outline feature to obtain an image matrix; S400: applying global averaging and maximum pooling to the image matrix to obtain a first tree outline visual embedding value;

[0054] S500: taking the remaining images in the first image set as test images, and extracting the second tree outline features;

[0055] S600: Calculate the first tree outline visual embedding value and the second tree outline feature to obtain a first pixel coordinate;

[0056] S700: input the sample image and the first pixel coordinates into the prediction model, and output the first mask; S800: if there are multiple tree segmentation tasks, determine the intersection-over-union ratio of the first mask and the test image mask, perform multiple predictions based on the determination results, and output all masks;

[0057] S900: Obtain tree outline features based on the output mask to calculate the true height of the tree. It should be noted that the performance of deep learning models depends largely on the quantity and quality of training data. When performing tree extraction tasks under specific tree species or specific environmental conditions, a large amount of labeled data may be required to train the model. The amount of training data obtained and the number of annotations are too large, resulting in too much time and energy spent in the early stage. At the same time, the model may not be able to maintain consistent high accuracy when facing a variety of actual application scenarios; and when the image contains multiple trees, the extraction accuracy will be significantly reduced, making it difficult to cope with complex multi-tree scenes.

[0058] Therefore, through steps S100-S900, a method for segmenting and extracting features of single trees and multiple trees is proposed through a single sample input, wherein information complementation can be achieved by applying global average pooling (GAP) and global maximum pooling (GMP) to the image matrix. GAP retains global statistical information by calculating the average value of each channel in the feature map, which helps to capture the overall distribution and trend of the features; GMP retains the most significant local features by selecting the maximum value of each channel in the feature map, which helps to capture the key details and prominent parts in the image. At the same time, the diversity of the model is increased, the generalization ability is improved, and overfitting is reduced. By making multiple predictions for the multi-tree segmentation task to improve the segmentation accuracy, the prediction strategy of the model is dynamically adjusted according to the IoU evaluation results, which can ensure that the final output mask is more accurate.

[0059] Example 2

[0060] Reference Figure 1-Figure 4Table 1 is an embodiment of the present invention, which provides a tree remote sensing image processing method based on one-time sample learning. In the implementation mode of the present application, the tree image acquired in step S100 can be collected by a calibrated camera; specifically, the following steps A1-A2 are included:

[0061] In A1, the camera that collects the image is calibrated to obtain the intrinsic parameter matrix and distortion coefficient, which specifically includes steps A11-A14:

[0062] A11: Collect chessboard images: Use A4 paper to print the chessboard calibration plate, where the chessboard specification is 9 rows and 9 columns; use the camera to collect ten chessboard images taken from different angles.

[0063] A12: Corner detection: Convert the captured image into a grayscale image, use the corner detection algorithm to detect the corners in the grayscale image, and extract and mark all detected corners.

[0064] Specifically, A12 includes:

[0065] A121: Calculate the gradient of the grayscale image and obtain the gradient value I of each pixel in the x and y directions x , I y ;

[0066] I x =Sobel x *I

[0067] I y =Sobel y *I

[0068] Among them, I represents the original image, Sobel x It represents that the Sobel operator performs edge detection on the image I in the x direction, and Sobely represents that the Sobel operator performs edge detection on the image I in the y direction.

[0069] A122: Calculate the autocorrelation matrix through the pixel gradients of the local window and weight the gradients to obtain the autocorrelation matrix M.

[0070]

[0071] Among them, I x (x+u,y+v),I y (x+u,y+v) represents the pixel intensity in the x and y directions (x+u,y+v) of the image respectively, and w(u,v) represents the weight function.

[0072] A123: Compute the corner response function C using the eigenvalues ​​of the autocorrelation matrix M f .

[0073] C f =det(M)-e·trace 2 (M)

[0074] Among them, det(M)=λ1λ2 is the determinant of the matrix M, trace(M)=λ1+λ2 is the trace of the matrix M, λ1 and λ2 are the eigenvalues ​​of M, and e is an empirical constant.

[0075] A124: Filter pixels with larger response function values ​​as candidate corner points, and then eliminate redundant corner points through non-maximum suppression to finally obtain the corner point set C. P .

[0076] A13: Get the camera internal parameter matrix and distortion coefficient. First, set the initial internal parameter matrix K and distortion coefficient k;

[0077]

[0078] k=(k1,k2,k3)

[0079] Among them, f x 、f y , C x , C y , k1, k2, and k3 are unknown parameters.

[0080] Afterwards, an iterative optimization algorithm is used to solve the internal parameter matrix, external parameter matrix and distortion coefficient, which can be expressed as:

[0081]

[0082] Among them, C P is a set of corner points, N is the total number of points, distort is the distortion correction function, K is the initial intrinsic parameter matrix, R, t are the extrinsic parameter matrices of the camera, which respectively describe the rotation and translation of the camera with respect to the world coordinate system, and π() is a function that returns the projection point of a point on the image plane.

[0083] A14: Calculate the accuracy of the calibration result through the back-projection error.

[0084] Specifically, A14 includes:

[0085] A141: Use the external parameter matrix to transform point P i Transform to the camera coordinate system.

[0086] P′ i =R i P i +t i

[0087] Among them, R i represents the rotation matrix, ti represents the translation vector, P′ i Indicates the new position after the transformation.

[0088] A142: Point P' in the camera coordinate system i Projected onto the normalized plane. Among them, point P′ i The coordinates of [X′ i ,Y′ i ,Z′ i ] is used to represent the matrix, where T represents the transpose of the matrix.

[0089]

[0090] Among them, point P′ i The coordinates of [X′ i ,Y′ i ,Z′ i ] is used to represent the matrix, where T represents the transpose of the matrix.

[0091] A143: Apply the distortion model to correct the point P on the normalized image plane norm,i , get the point P after distortion correction distorti .

[0092] A144: Use the intrinsic matrix to correct the distortion point P distorti Project back to the image plane.

[0093] P′ i =KP distorti

[0094] Where K represents the internal parameter matrix, where point P′ i The coordinates of [X′ i ,Y′ i ,Z′ i ], T represents the transpose of the matrix, and represents the projected point.

[0095] A145: Compute back-projection error via Euclidean distance.

[0096]

[0097] Among them, P′ i represents the projection point, P i represents a 3D point, N represents the number of all Euclidean distances, and Rep represents the back-projection error.

[0098] In A2, tree images are collected and a corrected first image set is obtained through distortion correction. Specifically, steps A21-A23 are included:

[0099] A21: Use a calibrated camera to collect tree images as original image data;

[0100] A22: Use a rangefinder to measure the distance between the shooting point and the tree, so as to be used in the subsequent calculation of the tree height; A23: Correct the image. First, use the inverse transformation to remove the image distortion through the distortion coefficient;

[0101] p undistorted =undistort(p,k)

[0102] Among them, p is a point on the original image and k is the distortion coefficient.

[0103] Afterwards, the points are projected back to the image plane using the intrinsic parameter matrix to obtain the rectified image.

[0104]

[0105] Where K is the internal parameter matrix.

[0106] In an optional implementation, the tree image obtained in step S100 may also be obtained by using 3D point cloud data obtained by ground-based laser radar (LiDAR), including 2D images and density information of trees. In another optional implementation, the tree image obtained in step S100 may also be synthesized by using image generation technology (such as GANs), which can increase the diversity and quantity of data and improve the generalization ability of the model.

[0107] In the embodiment of the present application, in step S200, any one of the first image sets is selected as the sample image S R , perform tree annotation and obtain the mask image S M , including the following steps B1-B3:

[0108] B1: Perform the first judgment on the selected sample image;

[0109] B2: If it is judged to be a single tree segmentation task, the single tree in the center of the image is marked to obtain a mask image containing a single tree;

[0110] B3: If it is determined to be a multi-tree segmentation task, all trees with clear structures in the image are labeled to obtain a mask image containing multiple trees.

[0111] It should be noted that the mask image S M Includes a mask image containing a single tree and a mask image containing multiple trees.

[0112] In the implementation manner of the present application, an interactive semi-automatic image annotation tool ISAT based on the Segment Anything Model may be used for annotation.

[0113] In an optional implementation, an automated labeling tool (such as a deep learning model) may be used to perform preliminary labeling on the trees.

[0114] In another optional implementation, a small amount of high-quality manually labeled data and a large amount of unlabeled data may be combined to perform labeling through a semi-supervised learning algorithm.

[0115] In the embodiment of the present application, the first tree outline feature can be obtained by extracting the sample image feature in step S200 by using the SAM pre-trained image encoder, and then extracting the sample image S R , get the first tree contour feature V f ; Among them, S R ∈R h*w*1 ,V f ∈R h*w*w ;

[0116] Specifically, the tree image can be a 256*256 RGB image. R represents a matrix, h represents a height dimension of 256, w represents a width dimension of 256, and c represents a channel number of 3;

[0117] V f =EncP(S R )

[0118] Among them, EncP stands for image encoder.

[0119] In an optional implementation, a U-Net model may be used to extract features of an image and generate a high-resolution segmentation mask, or a Mask R-CNN model may be used to detect objects in an image and generate a pixel-level mask for each object.

[0120] In the embodiment of the present application, the mask image is fused with the first tree outline feature in step S300 to obtain an image matrix, which can be converted into the mask image S by spatial multiplication. M With the first tree outline feature V f The pixel values ​​of the two are combined to obtain the image matrix M that combines the pixel values ​​of the two p , where M p ∈R h*w*c .

[0121] It should be noted that the pixel values ​​within the tree outline are affected by the mask, while the pixel values ​​outside the tree outline are suppressed to varying degrees. p It can be expressed as:

[0122] M p [i,j,l]=S M [i,j,l]×V f [i,j,l]

[0123] In an optional implementation, in step S300 , the mask image and the first tree outline feature may be fused by weighted summation to obtain an image matrix.

[0124] In another optional implementation, in step S300 , channel stitching may be used to fuse the mask image with the first tree contour feature to obtain an image matrix.

[0125] In the implementation manner of the present application, in step S400, global averaging and maximum pooling are applied to the image matrix to obtain a first tree outline visual embedding value, including the following steps C1-C3:

[0126] C1: Apply global average pooling to the image matrix, average all elements in each channel, and get the average output; the output of each channel is the first output;

[0127] Specifically, for the three channels R, G, and B, all elements in each channel are averaged to obtain a 1×1×c(3) output. The first output is specifically: for the i-th channel (i ranges from 1 to c), its output y avg,i It can be calculated by the following formula:

[0128]

[0129] Among them, x j,l,i It is the element of the i-th channel, j-th row, and l-th column in the input tensor, where h represents the height dimension of 256 and w represents the width dimension of 256.

[0130] C2: Apply global maximum pooling to the image matrix, find the maximum value of all elements in each channel, and get the maximum output; the output of each channel is the second output;

[0131] Specifically, the maximum value of all elements in each channel is calculated, and a 1×1×c output is obtained. The second output is specifically: for the i-th channel, its output y max,i It can be calculated by the following formula:

[0132]

[0133] C3: Combine the first output with the second output by weighted summation to obtain the first tree outline visual embedding value V e , where the global average pooling and global maximum pooling weight coefficients are the same.

[0134] Specifically, V e ∈R 1*256 ; Use the weighted summation method to define the weight coefficient α of GAP as 0.5, the weight of GMP as 1-α=0.5, and the final output V eThe calculation formula is:

[0135] V e =α×y avg,i +(1-α)×y max,i

[0136] In the present invention, the global average pooling GAP and the global maximum pooling GMP are considered to be equally important; the α parameter is set to 0.5, which means that the global average pooling GAP and the global maximum pooling GMP contribute the same degree.

[0137] In an optional implementation, global average pooling may be selected for operation in step S400. In another optional implementation, the contribution of each pooling may be controlled by changing the value of α in step S400.

[0138] In the implementation manner of the present application, in step S500, the remaining images in the first image set are used as test images to extract the second tree outline features;

[0139] Specifically, the test image T is extracted by an image encoder based on the SAM model i The second tree outline feature V t ,in It can be expressed by the following formula:

[0140] V t =EncP(T i )

[0141] Among them, EncP represents the image encoder of the SAM model:

[0142] In the implementation manner of the present application, the calculation in step S600 in combination with the first tree outline visual embedding value and the second tree outline feature to obtain the first pixel coordinate includes the following steps D1-D2:

[0143] D1: For each c-dimensional vector in the second tree outline feature, calculate the cosine similarity cos ij ; D2: Select cosine similarity cos ij The pixel coordinate of the highest point is the first pixel coordinate P max , as a hint input point for the prediction model, this point represents the area where the tree is most likely to be.

[0144] Specifically, cosine similarity cos ij It is expressed as:

[0145]

[0146] Among them, f ijk Indicates V t , that is: the second tree outline feature; vk V e , that is: the first tree outline visual embedding value.

[0147] So far, the reference image S is obtained R and point P max as the prompt input of the model. In an optional embodiment, in step S600, the similarity between the first tree outline visual embedding value and the second tree outline feature can also be measured by Euclidean distance to obtain the first pixel coordinate. In another optional embodiment, in step S600, the correlation between the first tree outline visual embedding value and the second tree outline feature can also be measured by using mutual information to obtain the first pixel coordinate. In the embodiment of the present application, in step S700, the sample image and the first pixel coordinate are input into the prediction model, and the first mask is output, including the following steps E1-E3:

[0148] E1: Introducing the attention mechanism into the prediction model, extracting attention features, and obtaining the first prediction feature;

[0149] E2: Input the first predicted feature into the decoding layer to obtain a preliminary mask;

[0150] E3: Perform light fine-tuning on the preliminary mask and output the first mask.

[0151] It should be noted that in order to allow the model to focus more attention on the target foreground area, namely the tree area, this method introduces an attention mechanism to help the model identify and segment trees more accurately and reduce the interference of background noise.

[0152] Specifically, the structure of the attention feature extraction layer is as follows Figure 3 As shown in Figure 1, it is mainly composed of two transformer stacks. Each transformer contains a self-attention module and two cross-attention modules, and the final input is the fused tokens prediction result hs∈R 1*7*256 and image features src∈R 1*4096*256. The first transformer layer fuses the tokens and tree features for the first time. The self-attention module learns the correlation between tokens features and outputs the attention result as the new prompt feature Q. The first cross-attention module integrates the tokens features with the image features to obtain new attention features. The multi-layer perceptron MLP is used to fuse the features of Q using a fully connected layer, and enters the second cross-attention module to integrate the image features with the tokens features. So far, the tokens features and image features have been fused for the first time respectively. Enter the second transformer and fuse the new features for the second time. Enter the last attention mechanism layer, associate and fuse the image features with the tokens features to obtain the first prediction features, including: the final prediction target, prompt features (1x7x256), image and mask features (1x4096x256).

[0153] Then, in step E2, the decoding layer is entered to obtain the mask as the output result. The decoding layer structure is as follows Figure 4 As shown in the figure, specifically, the prompt feature is input as a two-dimensional tensor, and the quality score of the mask (1x4) is obtained through three layers of full connection. The second to fifth feature dimensions are selected from the predicted target features for mask processing, and then three layers of full connection are used to obtain multiple sets of image feature weights (4x32). The image features are deconvolved and upsampled to obtain the final image features (32x256x256), and the image is weighted and summed by the feature weights to obtain the output preliminary mask.

[0154] It should be noted that in the present invention, it is found that for simple images containing only a single tree, satisfactory segmentation accuracy can be obtained through the above steps. However, for images containing complex scenes, such as rich vegetation, a large number of trees, and mutual occlusion, the segmentation accuracy will drop significantly. In order to ensure the segmentation accuracy, a lightweight fine-tuning mechanism is introduced to ensure that target objects with hierarchical structures can also be correctly segmented. Specifically, in step E2, after weighted summation of the image by feature weights, three groups of segmentation masks (1xnx256x256) are obtained, corresponding to the whole, part and sub-part of the tree, respectively, and the latter three masks are selected as preliminary masks. Then, the optimal mask, i.e., the first mask, is obtained by lightweight fine-tuning parameters. Specifically, in step E3, w1=1 / 3, w2=1 / 3, and w3=1-w1-w2=1 / 3 are set as the initial weight values ​​of the three masks, respectively. Perform a preliminary prediction, use the point with the highest similarity as the prompt input point, use the frozen SAM model, use the weighted summation method to get the predicted mask, and calculate the bounding rectangle of the mask greater than 0 as the first predicted rectangular box. Perform the first fine-tuning, use the point with the highest similarity and the first predicted rectangular box as prompt data, use the frozen model to make a prediction, select a mask based on the optimal score, calculate the bounding rectangle greater than 0, and use it as the first fine-tuning rectangular box. Perform the second fine-tuning, use the point with the highest similarity and the first fine-tuning rectangular box as prompt data, use the frozen model to make a prediction, and select the optimal mask based on the score, that is, the first mask.

[0155] In an optional implementation, in step S700, the sample image and the first pixel coordinates are input into the prediction model, and the first mask is output. The segmentation result can also be improved by introducing a conditional random field (CRF) layer into the prediction model and utilizing the global optimization capability of the CRF.

[0156] In another optional implementation, in step S700, the sample image and the first pixel coordinates are input into the prediction model, and the first mask is output. The segmentation result can also be improved by introducing a multi-scale fusion mechanism into the prediction model and fusing feature maps of different scales.

[0157] In the implementation manner of the present application, if there are multiple tree segmentation tasks in step S800, the intersection-over-union ratio of the first mask and the test image mask is determined, multiple predictions are performed based on the determination results, and all masks are output, including the following steps F1-F3:

[0158] F1: If the intersection-union ratio is greater than the first threshold, it is considered as a repeated mask and is not saved;

[0159] F2: Perform the next round of prediction until the number of masks reaches the maximum limit and the prediction ends; F3: Output all saved masks.

[0160] Specifically, the mask area in the test image is set to black, and the prediction is continued. The intersection over union (IoU value) of the first mask and the test image mask is calculated. If the IoU value is greater than the first threshold Nu, it is regarded as a repeated mask and is not saved. The next prediction is performed until the number of masks reaches the maximum limit Max, and the prediction is ended. All saved masks are output.

[0161] In the embodiment of the present application, Nu is set to 0.8, and Max is set to 10. When Nu>0.8, the two masks are considered to be repeated.

[0162] It should be noted that the first threshold Nu and the maximum limit Max are adjustable. In practice, the value of Nu can be adjusted to control the determination of repeated masks. For completeness, Table 1 provides data for verifying this parameter on a custom data set.

[0163] Table 1 Threshold setting verification table

[0164] N Number of correctly segmented images Total number of images error 0.5 30 50 0.4 0.6 37 50 0.26 0.7 45 50 0.9 0.8 48 50 0.96 0.9 48 50 0.96 0.95 48 50 0.96

[0165] Table 1 intuitively shows that, on the custom dataset, Nu exceeding 0.8 has achieved the best segmentation effect. In the present invention, since the number of trees in each image in the custom dataset does not exceed 10, the maximum limit is set to 10. In practice, the value of Max can be adjusted according to the actual situation to achieve the best segmentation effect.

[0166] In the embodiment of the present application, the tree outline features are obtained based on the output mask in step S800 to calculate the true height of the tree, including the following steps G1-G3:

[0167] G1: Extract all tree outline features in all output masks through edge detection method; G2: Select the largest outline among all tree outline features to obtain pixel height scale information; G3: Obtain the true height of the tree based on triangulated imaging calculation.

[0168] Specifically, the mask is first converted into a grayscale image, then Gaussian blur is used to reduce noise, then the Canny edge detection algorithm is used to find all contours, and finally, the maximum contour is found to obtain pixel height scale information. The tree height is calculated using the triangular imaging principle, which is expressed as:

[0169]

[0170] Among them, s represents the pixel size, d represents the distance between the camera and the object, h represents the height size information, f represents the focal length of the camera, and H is the actual height of the tree.

[0171] In an optional implementation, in step S800 , tree outline features are obtained based on the output mask, and the true height of the tree can also be calculated directly from the output mask by using a deep learning model.

[0172] In another optional implementation, in step S800 , tree outline features are obtained based on the output mask, and the actual height of the tree can also be estimated by the height of the point cloud data.

[0173] Example 3

[0174] Refer to Table 2-4 and Figure 5-Figure 6 Based on the previous embodiment, this embodiment provides an application case of a tree remote sensing image processing method based on one-time sample learning to illustrate the feasibility and beneficial effects of our solution.

[0175] In this embodiment, a Huawei Mate30 mobile phone is used as a camera for collecting images. Through the steps of the above embodiment S100, the internal parameter matrix K finally obtained by correction is:

[0176]

[0177] The distortion coefficient is:

[0178] (1:5201e-01, -6:0452e-01, -7:0468e-04, 2:7679e-03, 7:6273e-01) The back projection error is: 0:2665, it can be considered that the camera distortion is very small.

[0179] In the above step S400, global averaging and maximum pooling are applied to the image matrix to obtain a first tree outline visual embedding value, including the following steps C1-C3:

[0180] C1: Apply global average pooling to the image matrix, average all elements in each channel, and get the average output; the output of each channel is the first output;

[0181] C2: Apply global maximum pooling to the image matrix, find the maximum value of all elements in each channel, and get the maximum output; the output of each channel is the second output;

[0182] C3: Combine the first output with the second output by weighted summation to obtain the first tree outline visual embedding value V e , where the global average pooling and global maximum pooling weight coefficients are the same.

[0183] In the present invention, the global average pooling GAP and the global maximum pooling GMP are considered to be equally important; the weight parameters (a and 1-α) of the two are set to 0.5, which means that the global average pooling GAP and the global maximum pooling GMP have the same contribution.

[0184] Table 2 provides data on the mean values ​​of the IoU coefficients calculated on the custom tree dataset using different α parameters.

[0185] Table 2 Relationship between α parameter and average IoU coefficient

[0186] α 1-α IoU 0.1 0.9 91％ 0.3 0.7 90％ 0.5 0.5 93％ 0.7 0.3 89％ 0.9 0.1 92％

[0187] It can be seen from Table 2 that the value range is 0-1, which is an indicator of segmentation performance. The larger the value, the better. When the α value is 0.5, the IoU is the highest.

[0188] In the above step S700, the sample image and the first pixel coordinates are input into the prediction model, and the first mask is output, which includes the following steps E1-E3:

[0189] E1: Introducing the attention mechanism into the prediction model, extracting attention features, and obtaining the first prediction feature;

[0190] E2: Input the first predicted feature into the decoding layer to obtain a preliminary mask;

[0191] E3: Perform light fine-tuning on the preliminary mask and output the first mask.

[0192] Among them, the application effect of lightweight fine-tuning on complex images can be verified by Table X. Table 3 shows the IoU value comparison between the direct output mask and the output mask after fine-tuning parameters in ten groups of simple and complex images.

[0193] Table 3 Comparison of fine-tuning parameters and direct output

[0194]

[0195]

[0196] The results in Table 3 show that in simple tree images, the segmentation accuracy is slightly improved by adding fine-tuning parameters, but it does not play a big role. However, for tree images in complex scenes, the IoU value is significantly improved, indicating that this mechanism makes a huge contribution to the segmentation effect.

[0197] Based on the above steps S100 - S900 , Table 4 shows the tree heights obtained by using the method of this embodiment using ten trees as samples, and the errors between the tree heights and the actual tree heights.

[0198] Table 4 Prediction results

[0199] Tree No. Calculation height (m) Actual height (m) Relative error (%) 1 2.31 2.4 3.75 2 1.91 1.8 -6.11 3 2.54 2.6 2.30 4 2.97 3.2 7.18 5 2.33 2.2 -5.91 6 10.61 10.5 -1.05 7 11.35 12.0 5.42 8 9.42 8.9 -5.84 9 8.44 9.1 7.25 10 8.13 7.6 -6.97

[0200] It can be seen from Table 4 that the errors of the heights are all within an acceptable range, and the method of the present invention has a good prediction calculation effect.

[0201] at the same time, Figure 5 The visualization results of single tree segmentation are shown. The red color represents the segmented mask image. Figure 5 The visualization results show that single tree segmentation shows good results in both simple and complex environments. In complex environments, the trees in the center of the image are still automatically selected, which meets the expected requirements. Figure 6 The visualization results of multi-tree segmentation are shown. The left side is the original image, and the right side is the mask image of multiple trees cut out. Figure 6 The visualization results show that multi-tree segmentation still shows good performance in complex plant environments.

[0202] Example 4

[0203] The above is a schematic scheme of a tree remote sensing image processing method based on one-sample learning in this embodiment. It should be noted that the technical scheme of the tree remote sensing image processing system based on one-sample learning and the technical scheme of the tree remote sensing image processing method based on one-sample learning belong to the same concept. For details not described in detail in the technical scheme of the tree remote sensing image processing system based on one-sample learning in this embodiment, please refer to the description of the technical scheme of the tree remote sensing image processing method based on one-sample learning.

[0204] This embodiment also provides a system for a tree remote sensing image processing method based on one-time sample learning, including:

[0205] The acquisition module is used to acquire tree images and perform correction to obtain a first image set; the annotation module is used to select any one of the first image set as a sample image, perform tree annotation to obtain a mask image; and extract sample image features to obtain first tree outline features;

[0206] A first fusion module is used to fuse the mask image with the first tree outline feature to obtain an image matrix; a pooling module is used to apply global average and maximum pooling to the image matrix to obtain a first tree outline visual embedding value;

[0207] An extraction module, used for taking the remaining images in the first image set as test images, and extracting the second tree outline features;

[0208] A first calculation module, used to calculate by combining the first tree outline visual embedding value and the second tree outline feature to obtain a first pixel coordinate;

[0209] A prediction module, configured to input the sample image and the first pixel coordinates into a prediction model and output a first mask;

[0210] A judgment module, used for judging the intersection-and-union ratio of the first mask and the test image mask if there are multiple tree segmentation tasks, making multiple predictions based on the judgment results, and outputting all masks;

[0211] The second calculation module is used to obtain tree outline features based on the output mask to calculate the actual height of the tree.

[0212] This embodiment also provides an electronic device suitable for tree remote sensing image processing based on one-time sample learning, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the tree remote sensing image processing method based on one-time sample learning as proposed in the above embodiment.

[0213] This embodiment further provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for processing tree remote sensing images based on one-time sample learning as proposed in the above embodiment is implemented.

[0214] The storage medium proposed in this embodiment and the tree remote sensing image processing method based on one-time sample learning proposed in the above embodiment belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment. Through the above description of the implementation mode, the technicians in the relevant field can clearly understand that the present invention can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present invention.

[0215] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A tree remote sensing image processing method based on one-shot sample learning, characterized in that: include: Acquire tree images and perform correction to obtain a first image set; Select any one of the first image set as a sample image, perform tree annotation, and obtain a mask image; and extracting sample image features to obtain first tree outline features; The mask image is fused with the first tree contour feature to obtain an image matrix; Apply global average and maximum pooling to the image matrix to obtain the first tree outline visual embedding value; The remaining images in the first image set are used as test images to extract the second tree contour features; Combining the first tree outline visual embedding value and the second tree outline feature to calculate, to obtain a first pixel coordinate; Inputting the sample image and the first pixel coordinates into the prediction model, and outputting a first mask; If there are multiple tree segmentation tasks, determine the intersection-over-union ratio of the first mask and the test image mask, make multiple predictions based on the determination results, and output all masks; The tree outline features are obtained based on the output mask to calculate the true height of the tree.

2. The tree remote sensing image processing method based on one-time sample learning according to claim 1, characterized in that: Apply global average and maximum pooling to the image matrix to obtain the first tree outline visual embedding value, including: Apply global average pooling to the image matrix, average all elements in each channel, and obtain the average output; where the output of each channel is the first output; Apply global maximum pooling to the image matrix, find the maximum value of all elements in each channel, and obtain the maximum output; where the output of each channel is the second output; The first output is combined with the second output by weighted summation to obtain a first tree silhouette visual embedding value, wherein the global average pooling and the global maximum pooling weight coefficients are the same.

3. The tree remote sensing image processing method based on one-time sample learning according to claim 2, characterized in that: The first tree outline visual embedding value and the second tree outline feature are combined to calculate and obtain the first pixel coordinate, including: For each c-dimensional vector in the second tree outline feature, calculate the cosine similarity; The pixel coordinates of a point with the highest cosine similarity are selected as the first pixel coordinates, which are used as the input point for the prediction model prompt.

4. The tree remote sensing image processing method based on one-time sample learning according to claim 3, characterized in that: The sample image and the first pixel coordinate are input into the prediction model, and the first mask is output, including: Introducing an attention mechanism into the prediction model, performing attention feature extraction, and obtaining a first prediction feature; Input the first predicted feature into the decoding layer to obtain a preliminary mask; Perform light fine-tuning on the preliminary mask and output the first mask.

5. The tree remote sensing image processing method based on one-time sample learning according to claim 4, characterized in that: Select any one of the first image set as a sample image, perform tree annotation, and obtain a mask image, including: Performing a first judgment on the selected sample image; If it is determined to be a single tree segmentation task, the single tree in the center of the image is marked to obtain a mask image containing a single tree; If it is determined to be a multi-tree segmentation task, all trees with clear structures in the image are marked to obtain a mask image containing multiple trees.

6. The tree remote sensing image processing method based on one-shot sample learning according to claim 5, characterized in that: If there are multiple tree segmentation tasks, determine the intersection-over-union ratio of the first mask and the test image mask, make multiple predictions based on the judgment results, and output all masks, including: If the intersection-union ratio is greater than the first threshold, it is considered as a repeated mask and is not saved; Carry out the next round of prediction until the number of masks reaches the maximum limit, and then end the prediction; Output all saved masks.

7. The tree remote sensing image processing method based on one-shot sample learning according to claim 6, characterized in that: The tree outline features are obtained based on the output mask to calculate the true height of the tree, including: Through edge detection method, all tree contour features in all output masks are extracted; Select the largest contour among all tree contour features to obtain pixel height scale information; The true height of the trees is calculated based on triangulated imaging.

8. A system for tree remote sensing image processing method based on one-time sample learning according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used for acquiring tree images and performing correction to obtain a first image set; A labeling module is used to select any one of the first image set as a sample image, perform tree labeling, and obtain a mask image; and extract features of the sample image to obtain first tree outline features; A first fusion module, used for fusing the mask image with the first tree contour feature to obtain an image matrix; A pooling module, for applying global average and maximum pooling to the image matrix to obtain a first tree contour visual embedding value; An extraction module, used for taking the remaining images in the first image set as test images, and extracting the second tree outline features; A first calculation module, used to calculate by combining the first tree outline visual embedding value and the second tree outline feature to obtain a first pixel coordinate; A prediction module, configured to input the sample image and the first pixel coordinates into a prediction model and output a first mask; A judgment module, used for judging the intersection-and-union ratio of the first mask and the test image mask if there are multiple tree segmentation tasks, making multiple predictions based on the judgment results, and outputting all masks; The second calculation module is used to obtain tree outline features based on the output mask to calculate the actual height of the tree.

9. An electronic device, comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the tree remote sensing image processing method based on one-time sample learning as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the tree remote sensing image processing method based on one-time sample learning as claimed in any one of claims 1 to 7.

Images