City sidewalk tree three-dimensional form parameter calculation method and device based on street view image

Abstract

The application relates to a kind of city street tree three-dimensional form parameter calculation method and device based on street view image, wherein, method includes: robust visual features are extracted from street view image using depth estimation base model and visual feature extraction network layer;Robust depth features are obtained from street view image with the aid of depth estimation base model and depth feature extraction network layer, without fixed priori and artificial explanation, the accuracy of three-dimensional form calculation is improved;A three-dimensional form regression model is constructed, the interaction of visual and depth information is realized through attention layer, and tree mask regression and distance regression auxiliary tasks are introduced, focusing on the visual shape and spatial position of tree, to obtain the estimation result of the form parameter of tree. Thus, by constructing an end-to-end deep learning network model, the three-dimensional form parameters of street trees can be directly predicted when given street view images, greatly improving the prediction accuracy of the model and meeting the needs of rapid monitoring and management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and street view image application, and particularly relates to a method and device for calculating three-dimensional form parameters of urban street trees based on street view images. BACKGROUND

[0002] Street trees refer to tree species planted on both sides of roads and in the median strip, providing shade for vehicles and pedestrians and constituting the street view. Street trees play a key role in maintaining carbon-oxygen balance, regulating urban climate, improving environmental quality and protecting biodiversity. Systematic and accurate measurement of the form of street trees is of great significance to urban greening management and urban carbon storage estimation.

[0003] Traditional measurement methods rely on professional personnel using tape measures and altimeters for field surveys. Although the accuracy is high, a large amount of manpower and time is required, and the measurement efficiency is low. On the other hand, airborne or vehicle-mounted laser radar sensors can quickly scan and capture the three-dimensional structure information of street trees, but the high cost of data acquisition limits the feasibility of large-scale application, and is only suitable for small-scale plot studies.

[0004] In recent years, street view images have gradually become an important data source for urban research as a low-cost and widely covered observation method. Many researchers have begun to use street view images to assess some physical properties of street trees, such as species and location. However, existing street view image-based tree form measurement methods still have many limitations. First, many methods rely on fixed scale priors, such as assuming that the height and angle of the camera are known, which limits their applicability and generalization ability in different scenarios. Second, these methods usually only rely on visual cues for form measurement, which is easily affected by complex urban environments, thereby reducing the measurement accuracy. In addition, existing image processing methods are inefficient in processing large-scale data, making it difficult to meet the needs of rapid monitoring and management.

[0005] In summary, existing technologies excessively rely on fixed scale priors or visual cues for form measurement, which greatly affects the measurement accuracy of three-dimensional form parameters such as height and diameter at breast height of urban street trees, making it difficult to meet the needs of rapid monitoring and management, and thus need to be solved. SUMMARY

[0006] The present application provides a method and device for calculating three-dimensional form parameters of urban street trees based on street view images to solve the problem that existing technologies excessively rely on fixed scale priors or visual cues for form measurement, which greatly affects the measurement accuracy of three-dimensional form parameters such as height and diameter at breast height of urban street trees, making it difficult to meet the needs of rapid monitoring and management.

[0007] The first aspect embodiment of the present application provides a city street tree three-dimensional form parameter calculation method based on a street view image, comprising the following steps: obtaining a street view image of a target city street tree, and inputting the street view image into an encoder of a pre-constructed depth estimation base model to obtain corresponding multi-scale visual features, and obtaining target visual features corresponding to the multi-scale visual features through a pre-set visual feature extraction network layer and a residual connection layer; inputting the multi-scale visual features into a decoder of the depth estimation base model to extract implicit features corresponding to the street view image, and obtaining target depth features corresponding to the implicit features by using a pre-set depth feature extraction network layer and the residual connection layer; generating a tree boundary box corresponding to each street tree in the street view image by using a pre-set tree target detection algorithm, and calculating a boundary box embedding of a tree to be queried and a query vector corresponding to the boundary box embedding according to each tree boundary box and the target visual features, and inputting the boundary box embedding, the query vector, the target visual features and the target depth features into a pre-constructed three-dimensional form regression model to output three-dimensional form parameters of the tree to be queried in the street view image.

[0008] Optionally, in an embodiment of the present application, the street view image is input into the encoder of the pre-constructed depth estimation base model to obtain the corresponding multi-scale visual features, and the target visual features corresponding to the multi-scale visual features are obtained through the pre-set visual feature extraction network layer and the residual connection layer, comprising: constructing the depth estimation base model through the decoder, the encoder, the visual feature extraction network layer, the depth feature extraction network layer, the residual connection layer and a pre-set feedforward network; training the depth estimation base model, and dividing the street view image into a plurality of sub-image blocks by using the trained depth estimation base model, and extracting multi-scale visual features corresponding to the plurality of sub-image blocks through a pre-set self-attention mechanism; calculating a position embedding of each sub-image block in the plurality of sub-image blocks by using a pre-set position encoding strategy, and inputting the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain visual enhanced features corresponding to the multi-scale visual features; updating the visual enhanced features through the feedforward network and the residual connection layer to obtain the target visual features.

[0009] Optionally, in an embodiment of the present application, the inputting the multi-scale visual features into the decoder of the depth estimation base model to extract the implicit features corresponding to the street view image, and obtaining the target depth features corresponding to the implicit features by using the preset depth feature extraction network layer and the residual connection layer, comprises: performing data preprocessing on a preset optimization data set, and fine-tuning the decoder by using the optimization data set after data preprocessing; extracting the implicit features of the street view image by using the fine-tuned decoder, and extracting the depth enhanced features corresponding to the implicit features by using the depth feature extraction network layer; and updating the depth enhanced features by using the residual connection layer to generate the target depth features.

[0010] Optionally, in an embodiment of the present application, the generating the tree bounding box corresponding to each street tree in the street view image by using the preset tree target detection algorithm, and calculating the bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding according to each tree bounding box and the target visual features, and inputting the bounding box embedding, the query vector, the target visual features and the target depth features into the pre-constructed three-dimensional morphology regression model to output the three-dimensional morphology parameters of the tree to be queried in the street view image, comprises: generating the tree bounding box corresponding to each street tree in the street view image by using the tree target detection algorithm, and calculating the bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding according to the tree bounding box and the target visual features; constructing the three-dimensional morphology regression model based on a preset cross-attention layer, a self-attention layer and a multi-layer perceptron; and inputting the target visual features, the target depth features, the bounding box embedding corresponding to the tree to be queried and the query vector into the three-dimensional morphology regression model to calculate the three-dimensional morphology parameters of the tree to be queried by using the multi-layer perceptron and a preset tree mask regression and distance regression guidance model regression.

[0011] Optionally, in an embodiment of the present application, the mathematical expression of the position encoding strategy is:

[0012]

[0013] wherein, pe i,2t represents the 2tth element of the ith position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() represent the sine function and the cosine function respectively.

[0014] The second aspect embodiment of the present application provides a device for calculating three-dimensional morphological parameters of urban street trees based on street view images, comprising: a visual feature extraction module configured to obtain a street view image of a target urban street tree, input the street view image into an encoder of a pre-constructed depth estimation base model to obtain corresponding multi-scale visual features, and obtain target visual features corresponding to the multi-scale visual features through a pre-set visual feature extraction network layer and a residual connection layer; a depth feature extraction module configured to input the multi-scale visual features into a decoder of the depth estimation base model to extract implicit features corresponding to the street view image, and obtain target depth features corresponding to the implicit features by using a pre-set depth feature extraction network layer and the residual connection layer; and a query module configured to generate a tree bounding box corresponding to each street tree in the street view image by using a pre-set tree target detection algorithm, calculate a bounding box embedding of a tree to be queried and a query vector corresponding to the bounding box embedding according to each tree bounding box and the target visual features, and input the bounding box embedding, the query vector, the target visual features, and the target depth features into a pre-constructed three-dimensional morphological regression model to output three-dimensional morphological parameters of the tree to be queried in the street view image.

[0015] Optionally, in an embodiment of the present application, the visual feature extraction module comprises: a first modeling unit configured to construct the depth estimation base model by using the decoder, the encoder, the visual feature extraction network layer, the depth feature extraction network layer, the residual connection layer, and a pre-set feedforward network; a training unit configured to train the depth estimation base model, and divide the street view image into a plurality of sub-image blocks by using the trained depth estimation base model, and extract multi-scale visual features corresponding to the plurality of sub-image blocks by using a pre-set self-attention mechanism; a first calculation unit configured to calculate a position embedding of each sub-image block in the plurality of sub-image blocks by using a pre-set position encoding strategy, and input the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain visual enhanced features corresponding to the multi-scale visual features; and a first updating unit configured to update the visual enhanced features by using the feedforward network and the residual connection layer to obtain the target visual features.

[0016] Optionally, in an embodiment of the present application, the depth feature extraction module comprises: a fine-tuning unit configured to perform data preprocessing on a pre-set optimization data set, and fine-tune the decoder by using the optimization data set after data preprocessing; an extraction unit configured to extract implicit features of the street view image by using the fine-tuned decoder, and extract depth enhanced features corresponding to the implicit features by using the depth feature extraction network layer; and a second updating unit configured to update the depth enhanced features by using the residual connection layer to generate the target depth features.

[0017] Optionally, in an embodiment of the present application, the query module comprises: a generation unit configured to generate a tree bounding box corresponding to each street tree in the street view image by using the tree target detection algorithm, and calculate the bounding box embedding of the to-be-queried tree and a query vector corresponding to the bounding box embedding according to the tree bounding box and the target visual feature; a second modeling unit configured to construct the three-dimensional shape regression model based on a preset cross-attention layer, a self-attention layer and a multi-layer perceptron; and a second calculation unit configured to input the target visual feature, the target depth feature, the bounding box embedding corresponding to the to-be-queried tree and the query vector into the three-dimensional shape regression model, so as to regress and calculate the three-dimensional shape parameters of the to-be-queried tree by using the multi-layer perceptron and a preset tree mask regression and distance regression guidance model.

[0018] Optionally, in an embodiment of the present application, the mathematical expression of the position encoding strategy is:

[0019]

[0020] wherein, pe i,2t represents the 2tth element of the ith position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; and sin() and cos() represent the sine function and the cosine function, respectively.

[0021] The third aspect embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the street view image-based urban street tree three-dimensional shape parameter calculation method as described in the above embodiments.

[0022] The fourth aspect embodiment of the present application provides a computer readable storage medium, which stores a computer program executable by a processor to implement the street view image-based urban street tree three-dimensional shape parameter calculation method as described above.

[0023] The fifth aspect embodiment of the present application provides a computer program product comprising a computer program executable to implement the street view image-based urban street tree three-dimensional shape parameter calculation method as described above.

[0024] Therefore, the embodiments of the present application have the following beneficial effects:

[0025] The embodiment of the present application can obtain a street view image of a target city street tree, input the street view image into an encoder of a pre-constructed depth estimation base model to obtain corresponding multi-scale visual features, and obtain target visual features corresponding to the multi-scale visual features through a pre-set visual feature extraction network layer and a residual connection layer; input the multi-scale visual features into a decoder of the depth estimation base model to extract implicit features corresponding to the street view image, and obtain target depth features corresponding to the implicit features by using a pre-set depth feature extraction network layer and a residual connection layer; generate a tree boundary box corresponding to each street tree in the street view image by using a pre-set tree target detection algorithm, calculate a boundary box embedding of a tree to be queried and a query vector corresponding to the boundary box embedding according to each tree boundary box and the target visual features, and input the boundary box embedding, the query vector, the target visual features and the target depth features into a pre-constructed three-dimensional shape regression model to output three-dimensional shape parameters of the tree to be queried in the street view image. The method for efficiently and accurately calculating the three-dimensional shape parameters of the tree from the street view image has important scientific significance and application value for promoting urban greening management and ecological research. Thus, the problems of the prior art, such as excessive dependence on fixed scale priori or visual clues for shape measurement, great influence on the measurement accuracy of three-dimensional shape parameters such as the height and diameter at breast height of the urban street tree, and difficulty in meeting the needs of rapid monitoring and management, are solved.

[0026] Additional aspects and advantages of the present application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0027] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, wherein:

[0028] Figure 1 A flowchart of a method for calculating three-dimensional shape parameters of urban street trees based on street view images according to an embodiment of the present application is shown in (a) of FIG. 1;

[0029] Figure 2 A logic architecture schematic diagram of a method for calculating three-dimensional shape parameters of urban street trees based on street view images according to an embodiment of the present application is shown in (b) of FIG. 1;

[0030] Figure 3 A model architecture schematic diagram according to an embodiment of the present application is shown in (c) of FIG. 1;

[0031] Figure 3 A visual feature extraction model architecture schematic diagram according to an embodiment of the present application is shown in (a) of FIG. 2;

[0032] Figure 3(b) in FIG. 1 is a schematic diagram of a deep feature extraction model architecture provided by an embodiment of the present application;

[0033] Figure 3 (c) in FIG. 1 is a schematic diagram of a shape parameter regression calculation model architecture provided by an embodiment of the present application;

[0034] Figure 4 a schematic diagram of a data set composition provided by an embodiment of the present application;

[0035] Figure 5 a schematic diagram of a prediction result provided by an embodiment of the present application;

[0036] Figure 6 a schematic diagram of a city street tree three-dimensional shape parameter calculation device based on street view images according to an embodiment of the present application;

[0037] Figure 7 a schematic diagram of an electronic device provided by an embodiment of the present application.

[0038] wherein, 10 - a city street tree three-dimensional shape parameter calculation device based on street view images; 100 - a visual feature extraction module, 200 - a deep feature extraction module, 300 - a query module; 701 - a memory, 702 - a processor, 703 - a communication interface. DETAILED DESCRIPTION

[0039] Embodiments of the present application are described in detail below with reference to the accompanying drawings. Examples of the embodiments are shown in the drawings, in which the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0040] A method and device for calculating three-dimensional morphological parameters of urban street trees based on street view images are described below with reference to the accompanying drawings. To address the problems mentioned in the background art, the present application provides a method for calculating three-dimensional morphological parameters of urban street trees based on street view images. In this method, a street view image of a target urban street tree is obtained and input into an encoder of a pre-constructed depth estimation base model to obtain corresponding multi-scale visual features. A target visual feature corresponding to the multi-scale visual features is obtained through a pre-set visual feature extraction network layer and a residual connection layer. The multi-scale visual features are input into a decoder of the depth estimation base model to extract implicit features corresponding to the street view image. A target depth feature corresponding to the implicit features is obtained using a pre-set depth feature extraction network layer and a residual connection layer. A tree target detection algorithm is used to generate a tree bounding box corresponding to each street tree in the street view image. A bounding box embedding and a query vector corresponding to the bounding box embedding of a tree to be queried are calculated based on each tree bounding box and the target visual feature. The bounding box embedding, the query vector, the target visual feature, and the target depth feature are input into a pre-constructed three-dimensional morphological regression model to output three-dimensional morphological parameters of the tree to be queried in the street view image. The method of the present application can efficiently and accurately calculate three-dimensional morphological parameters of trees from street view images, which has important scientific significance and application value for promoting urban greening management and ecological research. Thus, the problems of excessive reliance on fixed scale priors or visual cues for morphological measurement, which greatly affects the measurement accuracy of three-dimensional morphological parameters such as height and diameter at breast height of urban street trees, and difficulty in meeting the needs of rapid monitoring and management, are solved.

[0041] Specifically, Figure 1 A flowchart of a method for calculating three-dimensional morphological parameters of urban street trees based on street view images is provided.

[0042] As Figure 1 shown, the method for calculating three-dimensional morphological parameters of urban street trees based on street view images includes the following steps:

[0043] In step S101, a street view image of a target urban street tree is obtained and input into an encoder of a pre-constructed depth estimation base model to obtain corresponding multi-scale visual features. A target visual feature corresponding to the multi-scale visual features is obtained through a pre-set visual feature extraction network layer and a residual connection layer.

[0044] The embodiment of the present application can first extract multi-scale visual features of the street view image through a self-attention mechanism by inputting the street view image into an encoder of a pre-trained depth estimation base model; meanwhile, the embodiment of the present application introduces a learnable visual transformer module to further enhance the visual features, and updates the input features using a residual connection, thereby obtaining robust visual features (i.e., target visual features).

[0045] Optionally, in an embodiment of the present application, the street view image is input into an encoder of a pre-constructed depth estimation base model to obtain corresponding multi-scale visual features, and the target visual features corresponding to the multi-scale visual features are obtained through a preset visual feature extraction network layer and a residual connection layer, including: constructing a depth estimation base model through a decoder, an encoder, a visual feature extraction network layer, a depth feature extraction network layer, a residual connection layer and a preset feedforward network; training the depth estimation base model, and dividing the street view image into a plurality of sub-image blocks using the trained depth estimation base model, and extracting multi-scale visual features corresponding to the plurality of sub-image blocks through a preset self-attention mechanism; calculating the position embedding of each sub-image block in the plurality of sub-image blocks using a preset position encoding strategy, and inputting the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain visual enhanced features corresponding to the multi-scale visual features; updating the visual enhanced features through the feedforward network and the residual connection layer to obtain the target visual features.

[0046] In actual execution, the embodiment of the present application can first input the street view image into an encoder of a pre-trained depth estimation base model (Depth Anything), which extracts multi-level visual features of the street view image through a self-attention mechanism, further enhances the visual features through a learnable visual transformer module, and updates the input features using a residual connection to obtain robust visual features (i.e., target visual features) F v As shown in Figure 2 .

[0047] As understood by those skilled in the art, since the visual base model (Depth Anything) is trained on a large amount of image data exceeding 60 million, it has strong generalization ability and can help the network to replace the traditional prior knowledge relying on artificial setting with strong data prior knowledge in the base model, and the embodiment of the present application can select the visual base model (Depth Anything) as the backbone network, as shown in Figure 3Depth Anything encoder is a Vision Transformer (ViT) model, which can first divide the input street view image into L image blocks (i.e., multiple sub-image blocks) with a size of 14x14; secondly, extract multi-level visual features (i.e., multi-scale visual features) through a series of stacked attention blocks wherein, represents the i-th level feature, H and W represent the height and width of the image respectively, and C represents the channel dimension.

[0048] Optionally, in an embodiment of the present application, the mathematical expression of the position encoding strategy is:

[0049]

[0050] wherein, pe i,2t represents the 2t-th element of the i-th position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() represent the sine function and the cosine function respectively.

[0051] For the multi-level feature F b , the higher the level is, the deeper the feature obtained is. In the specific implementation process, in order to exclude the influence of low-level features, the deepest feature F is used in the embodiments of the present application. wherein, represents the number of tokens after flattening, and C represents the channel dimension; the position embedding of each token is calculated while flattening The mathematical expression of the position encoding strategy is as follows:

[0052]

[0053] wherein, pe i,2t represents the 2t-th element of the i-th position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() represent the sine function and the cosine function respectively.

[0054] Secondly, the embodiments of the present application can input the feature tokens and the corresponding position embedding pe are input into a self-attention layer, and a calculation formula of the self-attention layer is represented as

[0055] A q ,A k = Linear(F) + PE, A v = Linear(F)

[0056]

[0057] wherein A q , A k , and A v respectively represent a Query matrix, a Key matrix, and a Value matrix; F is an input feature map; PE is an input position embedding; C is a channel dimension; Linear() is a linear transformation function, which can be represented as WF + b; and Softmax() is used for normalizing weights, so that the sum of all weights is 1.

[0058] Subsequently, the feature SA(F, PE) extracted through the self-attention layer is input into a feed forward network (FFN) composed of two linear layers and an activation function, and the feed forward network layer can be represented by the following formula:

[0059] FFN(F) = Linear(LeakyReLU(Linear(F)))

[0060] wherein F is an input feature map; Linear() is a linear transformation function; and LeakyReLU() is an activation function, which is used to introduce nonlinear features, and the definition of LeakyReLU() is shown in the following formula:

[0061]

[0062] Finally, the embodiment of the present application can update the input feature F to obtain a visual feature F v .

[0063] In summary, the embodiment of the present application obtains a mathematical expression of a robust visual feature by inputting a street view image into a pre-trained depth estimation base model, and combining a self-attention mechanism, a learnable visual transformer module, and a residual connection, as follows:

[0064]

[0065] It can be understood that the embodiment of the application first inputs the street view image into the pre-trained depth estimation base model Depth Anything, which divides the street view image into multiple image blocks through the self-attention mechanism and extracts the multi-level visual features of the street view image; secondly, the position embedding of each image block is calculated through the position encoding formula, and it is input into the learnable visual transformer module together with the features to further enhance the visual features; finally, the input features are updated through the feedforward network and the residual connection to obtain the robust visual features F v , so as to lay a foundation for subsequent depth feature extraction and three-dimensional shape regression while focusing on the domain knowledge of a specific task and utilizing the powerful generalization ability of the base model.

[0066] In step S102, the multi-scale visual features are input into the decoder of the depth estimation base model to extract the implicit features corresponding to the street view image, and the target depth features corresponding to the implicit features are obtained by using the preset depth feature extraction network layer and the residual connection layer.

[0067] Furthermore, the embodiment of the application also needs to extract the implicit features of the street view image by using the pre-trained Depth Anything model, and further extract the robust depth clues (i.e. target depth features) by fine-tuning and introducing the learnable depth transformer module to predict the measured depth.

[0068] Optionally, in an embodiment of the application, the multi-scale visual features are input into the decoder of the depth estimation base model to extract the implicit features corresponding to the street view image, and the target depth features corresponding to the implicit features are obtained by using the preset depth feature extraction network layer and the residual connection layer, including: performing data preprocessing on the preset optimization data set, and fine-tuning the decoder by using the optimization data set after data preprocessing; extracting the implicit features of the street view image by using the fine-tuned decoder, and extracting the depth enhanced features corresponding to the implicit features by using the depth feature extraction network layer; updating the depth enhanced features by using the residual connection layer to generate the target depth features.

[0069] It should be noted that the embodiment of the application can first fine-tune the Depth Anything decoder by using the WHU-RSTree data set; secondly, extract the implicit features of the street view image by using the Depth Anything decoder; then, further enhance the extracted depth features by using the learnable depth transformer module; finally, update the input features by using the residual connection to obtain the robust depth features (i.e. target depth features) F d .

[0070] As skilled in the art will appreciate, since most existing depth estimation models are trained in disparity space, they can only capture relative distance relationships between objects, lacking real scale information.

[0071] To achieve more accurate metric depth prediction, embodiments of the present application can fine-tune the Depth Anything decoder using the WHU-RSTree dataset in advance. The WHU-RSTree dataset is composed of point cloud data and 12447 panoramic images of about 68 kilometers in Nanjing, China, containing annotations of more than 20000 trees, including instance segmentation results, tree species and morphological parameters.

[0072] During the fine-tuning stage, embodiments of the present application can first use the spherical projection method to crop the panoramic images in the WHU-RSTree dataset. Specifically, since street trees are generally located on both sides of the road, 90° and 270° horizontal angles are selected as anchor points, the image size is cropped to 1024x1024, the horizontal and vertical field of view (FoV) is set to 90°, and images that do not contain valid street tree annotations are deleted, thereby obtaining 11011 street view images; secondly, based on the point cloud data in the WHU-RSTree dataset, a rough depth map is constructed by projection, the specific process is as follows:

[0073] 1. Project the point cloud within a 95m range from the camera center to the image and calculate the Z value as the depth value, taking into account the discrete nature of the point cloud which may lead to incorrect depth relationships, in order to improve the quality of the depth map;

[0074] 2. Remove outliers and use a multi-scale dilation convolution to complete the sparse depth into a dense depth map according to existing methods;

[0075] 3. Calculate the difference between the dense depth value and the original sparse depth value, where the smaller the difference, the higher the probability of becoming a foreground point. By setting a threshold for this difference, the depth points are filtered and the final depth map is obtained.

[0076] Further, embodiments of the present application can use existing methods to identify the sky area in the street view image, and forcibly require the depth of these areas to exceed a predefined maximum depth (which can generally be set to 80 meters), to ensure that the depth value of the far distance area is not underestimated, thereby improving the overall prediction accuracy; finally, embodiments of the present application can introduce constraints on surface discovery and confidence level scores to improve the accuracy of the metric depth D. It can be understood that surface discovery can help the model understand the directionality of the object surface, while the confidence level score can indicate the degree of confidence of the model in predicting the depth of a particular area, and these constraints help to improve the quality of the prediction results.

[0077] As shown in (b) in Figure 3 , the Depth Anything decoder is based on a DPT (Dense Prediction Transformer) architecture, which receives the features F b from different levels of the Depth Anything encoder as input and learns a multi-scale, high-level abstract representation, i.e., the implicit features f d of the street view image through a series of complex convolutional layers and attention mechanisms.

[0078] After that, the embodiment of the present application can flatten the feature map into feature tokens wherein, represents the number of flattened tokens, and C represents the channel dimension; while flattening, the position embedding of each token is calculated The calculation formula is as follows:

[0079]

[0080] wherein, pe i is the position embedding vector of the i-th feature token; pe i,2t represents the 2t-th element of the i-th position embedding vector pe i ; pe i,2t and pe i,2t+1 are two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() represent the sine function and the cosine function, respectively.

[0081] Further, the embodiment of the present application can input the feature tokens f d and the corresponding position embedding pe into a self-attention layer, and the calculation formula of the self-attention layer is represented as:

[0082] A q ,A k =Linear(F)+PE,A v =Linear(F)

[0083]

[0084] wherein, A q , A k , A vQuery, Key, Value respectively represent Query matrix, Key matrix, Value matrix; F represents input feature map; PE represents input position embedding; C represents channel dimension; Linear() represents linear transformation function, which can be represented as WF+b; Softmax() is used for normalizing weight, so that the sum of all weights is 1.

[0085] Subsequently, the feature SA(F, PE) extracted through the self-attention layer can be input into a feedforward network composed of two linear layers and an activation function. The feedforward network layer can be represented by the following formula:

[0086] FFN(F)=Linear(LeakyReLU(Linear(F)))

[0087] Wherein, F represents input feature map; Linear() represents linear transformation function; LeakyReLU() represents activation function, which is used to introduce nonlinear features, and the definition of LeakyReLU() is as follows:

[0088]

[0089] Finally, the embodiment of the present application can update the input feature f f by using residual connection to obtain F d . Specifically, the embodiment of the present application needs to first downsample the metric depth D by using the average pooling method to match the resolution of the feature f d . Then, the depth embedding DE of each feature token is calculated, and the calculation formula is as follows:

[0090]

[0091] Wherein, de i is the depth embedding vector of the i-th feature token; de i,2t represents the 2t-th element of the i-th depth embedding vector de i ; de i,2t and de i,2t+1 are two consecutive elements in the depth embedding vector de i ; d i is the average metric depth of each feature token; C is the channel dimension; sin() and cos() represent sine function and cosine function respectively.

[0092] It can be understood that the above sine encoding can effectively capture the difference between the depth values and improve the ability of the network to represent spatial information.

[0093] In summary, the embodiment of the present application extracts the implicit features of the street view image by using the pre-trained Depth Anything model, and predicts the metric depth by fine-tuning and introducing a learnable depth transformer module, and then extracts the mathematical expression of the robust depth clue,

[0094] F d = f d + FFN(SA(f d , PE)) + DE

[0095] Therefore, the embodiment of the present application develops a new tree-shaped data set TMD by post-processing the data of the WHU-RSTree data set to fill the gap in the field, thereby effectively supporting the shape analysis of roadside trees and promoting the development of the related research community.

[0096] In step S103, a tree target detection algorithm is used to generate a tree bounding box corresponding to each street tree in the street view image, and a bounding box embedding and a query vector corresponding to the bounding box embedding of the tree to be queried are calculated according to each tree bounding box and the target visual feature. The bounding box embedding, the query vector, the target visual feature and the target depth feature are input into a pre-constructed three-dimensional shape regression model to output the three-dimensional shape parameters of the tree to be queried in the street view image.

[0097] Further, the embodiment of the present application also needs to construct a three-dimensional shape regression model, which regards the tree as a query vector, stacks attention layers in the regression model to promote the deep interaction between visual and depth information, and introduces two auxiliary regression tasks of tree mask regression and distance regression guide model. One guide model focuses on the visual shape of the tree in the image, and the other emphasizes the spatial position of the tree in space to achieve efficient and accurate shape parameter estimation.

[0098] Optionally, in an embodiment of the present application, a preset tree target detection algorithm is used to generate a tree bounding box corresponding to each street tree in a street view image, and a bounding box embedding and a query vector corresponding to the bounding box embedding of a tree to be queried are calculated according to each tree bounding box and a target visual feature, and the bounding box embedding, the query vector, the target visual feature and a target depth feature are input into a pre-constructed three-dimensional shape regression model to output a three-dimensional shape parameter of the tree to be queried in the street view image, including: generating a tree bounding box corresponding to each street tree in a street view image by a tree target detection algorithm, and calculating a bounding box embedding and a query vector corresponding to the bounding box embedding of a tree to be queried according to the tree bounding box and a target visual feature; based on a preset cross-attention layer, a self-attention layer and a multi-layer perceptron, a three-dimensional shape regression model is constructed; the target visual feature, the target depth feature, the bounding box embedding corresponding to the tree to be queried and the query vector are input into the three-dimensional shape regression model, so as to use the multi-layer perceptron and the preset tree mask regression and distance regression guidance model to regress and calculate the three-dimensional shape parameter of the tree to be queried.

[0099] As shown in (c) in FIG. 1, Figure 3 To fuse the visual information of the bounding box, the embodiment of the present application first designs a new bounding box prompt embedding strategy to obtain the bounding box embedding be i and the query vector q i ; then, using the bounding box embedding be i , the query vector q i , the depth feature F d , the visual feature F v as input, the cross-attention layer and the self-attention layer are used to promote the model to pay more attention to the interaction of the depth information and the visual information of the tree in the bounding box; finally, the multi-layer perceptron and two auxiliary regression tasks are used to jointly regress and calculate the shape parameter p i ={dbh i ,th i}.

[0100] It should be noted that the embodiment of the present application does not directly detect the street view image, but uses the bounding box containing the tree detected by the target detection method as input, so as to benefit from the superior performance of the target detection model trained on a large-scale data set, and efficiently obtain more stable and accurate parameter estimation results.

[0101] Specifically, the embodiment of the present application takes a street view image as input, and uses an existing target detection method to obtain a plurality of bounding boxes b For each bounding box b i , the center coordinates thereof are used as anchor points, and the corresponding feature F v is extracted to initialize the query vector q The method of obtaining a query vector based on an anchor point effectively reduces the search space of the network, thereby facilitating faster and more accurate convergence.

[0102] Secondly, the height and width information of the bounding box can be encoded by the embodiment of the application to construct the bounding box embedding vector be i The specific process is as follows:

[0103] 1. The height and width embedding of each bounding box b i is calculated by using the formula, and the calculation formula is as follows:

[0104]

[0105] Wherein, he i and we i are the height and width embedding vectors of the i-th bounding box; he i,2t represents the 2t-th element of the height embedding vector he i of the i-th bounding box; he i,2t and he i,2t+1 are two consecutive elements in the embedding vector he i ; h i is the height of each bounding box; C is the channel dimension; sin() and cos() represent the sine function and the cosine function respectively, and the calculation formula of the width embedding vector is the same;

[0106] 2. The position embedding vector pe i corresponding to the query q i is calculated by using the formula, and three learnable parameters {E h ,E w ,E p} are introduced, E h ,E w ,E p are identifiers of height embedding, width embedding and position embedding respectively, and the introduction of the identifier helps the network to learn the specific representation of different elements and improve the ability to express different types of information;

[0107] 3. A linear function is used to fuse these embeddings to obtain the bounding box embedding be i , and the mathematical expression is as follows:

[0108] be i =Linear(Concatenate((he i +E h ),(we i +E w ),(pe i +E p )))

[0109] where Concatenate() represents a concatenation operation in the channel dimension, i.e., connecting different types of embeddings in the channel dimension.

[0110] After that, an embodiment of the present application can give a bounding box embedding be i and a corresponding query vector q i , while inputting the target visual feature F v and the target depth feature F d ; the three-dimensional morphological parameter regression model MPRM can realize the interaction of visual and depth information by using a cross-attention layer (CA) and a self-attention layer (SA), and the mathematical expression of the fusion process is:

[0111] A q = Linear(Q) + BE

[0112] A k , A v = Linear(F)

[0113]

[0114] Q = Q + FFN(CA(SA(CA(Q, BE, F d ), BE), BE, F v ))

[0115] where Q is a query vector; BE is a bounding box embedding vector; F is a feature map; C is a channel dimension; A q , A k , A v are query matrix, key matrix, and value matrix obtained by linear transformation Linear(), respectively; CA and SA represent cross-attention layer and self-attention layer, respectively; FFN represents a feed-forward neural network layer.

[0116] Through the above processing, the query vector can be updated and integrated with visual information and depth information; finally, the three-dimensional morphological parameter regression model MPRM can use a shared multi-layer perceptron to regress the morphological parameters p i ={dbh i , th i} of the tree and the distance dis i from the measured tree to the camera center based on the updated query vector Q and the original bounding box embedding BE; the design of the shared multi-layer perceptron can reduce the number of parameters, so that the model can complete the prediction of multiple targets in a single forward propagation, improving the calculation efficiency; in addition, in order to enable the model to regress the instance mask m iThe low-level visual features are combined with the visual features F v , and the results are up-sampled to the original resolution of the input image, and then multiplied with the query vector q i by a dot product operation to calculate the instance mask m corresponding to each tree through a linear layer. i As shown in Figure 4 , the calculation formula is as follows:

[0117]

[0118] wherein, represents a dot product operation, which is used to measure the similarity between the two, and the accurate mask of each tree instance can be obtained; the distance dis i and the instance mask m i are used to assist in training the network and guide the network to focus on the visual shape of the trees in the image and the spatial position in space, so as to achieve efficient and accurate parameter evaluation, and the prediction result is as shown in Figure 5 .

[0119] In summary, the embodiment of the present application first adopts a visual basic model as the backbone network. The visual basic model can replace the traditional prior knowledge relying on artificial setting by using its powerful data prior knowledge, so as to extract robust visual features; secondly, considering that depth is a robust and widely applicable data connecting 2D image and 3D world, which can not only provide scale prior but also complete the missing spatial information, therefore, a depth basic estimation model is used to extract robust depth clues by constructing a depth transformer model; finally, a morphological parameter regression model is developed to fuse visual and depth clues to calculate the morphological parameters of the trees.

[0120] Therefore, the embodiment of the present application realizes an end-to-end street image-based tree morphology measurement method by an end-to-end network for quickly and accurately estimating the morphology parameters of street trees from low-cost street images, without relying on fixed scale prior or artificial interpretation, by integrating visual and depth clues with the help of a depth estimation basic model, which greatly improves the efficiency of urban number management and vegetation carbon storage estimation.

[0121] According to the method for calculating the three-dimensional shape parameters of urban street trees based on street view images provided in the embodiments of the present application, robust visual features are extracted from street view images by using a depth estimation base model and a visual feature extraction network layer; robust depth features are obtained from the street view images by means of the depth estimation base model and the depth feature extraction network layer, without fixed priori and manual interpretation, so as to improve the accuracy of three-dimensional shape calculation; a three-dimensional shape regression model is constructed, the interaction of visual and depth information is realized through an attention layer, and a tree mask regression and distance regression auxiliary task is introduced to focus on the visual shape and spatial position of the trees, so as to obtain the estimation result of the shape parameters of the trees. The method for efficiently and accurately calculating the three-dimensional shape parameters of trees from street view images has important scientific significance and application value for promoting urban greening management and ecological research.

[0122] Secondly, the device for calculating the three-dimensional shape parameters of urban street trees based on street view images according to the embodiments of the present application is described with reference to the accompanying drawings.

[0123] Figure 6 FIG. 1 is a block schematic diagram of the device for calculating the three-dimensional shape parameters of urban street trees based on street view images according to the embodiments of the present application.

[0124] As shown in FIG. 1, the device for calculating the three-dimensional shape parameters of urban street trees based on street view images 10 comprises a visual feature extraction module 100, a depth feature extraction module 200 and a query module 300. Figure 6

[0125] The visual feature extraction module 100 is configured to acquire a street view image of a target urban street tree, input the street view image into an encoder of a pre-constructed depth estimation base model to obtain corresponding multi-scale visual features, and acquire target visual features corresponding to the multi-scale visual features through a pre-set visual feature extraction network layer and a residual connection layer.

[0126] The depth feature extraction module 200 is configured to input the multi-scale visual features into a decoder of the depth estimation base model to extract implicit features corresponding to the street view image, and acquire target depth features corresponding to the implicit features by using a pre-set depth feature extraction network layer and a residual connection layer.

[0127] The query module 300 is configured to generate a tree bounding box corresponding to each street tree in the street view image by using a pre-set tree target detection algorithm, calculate a bounding box embedding and a query vector corresponding to the bounding box embedding of a tree to be queried according to each tree bounding box and the target visual features, and input the bounding box embedding, the query vector, the target visual features and the target depth features into a pre-constructed three-dimensional shape regression model to output the three-dimensional shape parameters of the tree to be queried in the street view image.

[0128] ​Optionally, in an embodiment of the present application, the visual feature extraction module 100 comprises a first modeling unit, a training unit, a first calculation unit and a first updating unit.

[0129] The first modeling unit is configured to construct a depth estimation base model by using a decoder, an encoder, a visual feature extraction network layer, a depth feature extraction network layer, a residual connection layer and a preset feedforward network.

[0130] The training unit is configured to train the depth estimation base model, divide the street view image into a plurality of sub-image blocks by using the trained depth estimation base model, and extract a plurality of multi-scale visual features corresponding to the plurality of sub-image blocks by using a preset self-attention mechanism.

[0131] The first calculation unit is configured to calculate a position embedding of each of the plurality of sub-image blocks by using a preset position encoding strategy, and input the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain visual enhanced features corresponding to the multi-scale visual features.

[0132] The first updating unit is configured to update the visual enhanced features by using the feedforward network and the residual connection layer to obtain target visual features.

[0133] Optionally, in an embodiment of the present application, the depth feature extraction module 200 comprises a fine-tuning unit, an extraction unit and a second updating unit.

[0134] The fine-tuning unit is configured to perform data preprocessing on a preset optimization data set, and fine-tune the decoder by using the data-preprocessed optimization data set.

[0135] The extraction unit is configured to extract implicit features of the street view image by using the fine-tuned decoder, and extract depth enhanced features corresponding to the implicit features by using a depth feature extraction network layer.

[0136] The second updating unit is configured to update the depth enhanced features by using a residual connection layer to generate target depth features.

[0137] Optionally, in an embodiment of the present application, the query module 300 comprises a generation unit, a second modeling unit and a second calculation unit.

[0138] The generation unit is configured to generate a tree bounding box corresponding to each street tree in the street view image by using a tree target detection algorithm, and calculate a bounding box embedding of a to-be-queried tree and a query vector corresponding to the bounding box embedding according to the tree bounding box and the target visual features.

[0139] The second modeling unit is configured to construct a three-dimensional morphology regression model based on a preset cross-attention layer, a self-attention layer and a multi-layer perceptron.

[0140] The second calculation unit is configured to input the target visual feature, the target depth feature, the boundary box embedding corresponding to the tree to be queried, and the query vector into the three-dimensional shape regression model, so as to calculate the three-dimensional shape parameters of the tree to be queried by using the multilayer perceptron and the preset tree mask regression and distance regression guidance model.

[0141] Optionally, in an embodiment of the present application, the mathematical expression of the position encoding strategy is:

[0142]

[0143] wherein pe i,2t represents the 2tth element of the ith position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() represent the sine function and the cosine function respectively.

[0144] It should be noted that the foregoing explanation and description of the embodiment of the method for calculating the three-dimensional shape parameters of the urban street trees based on the street view images also applies to the embodiment of the device for calculating the three-dimensional shape parameters of the urban street trees based on the street view images, which will not be described herein again.

[0145] The device for calculating the three-dimensional shape parameters of the urban street trees based on the street view images according to the embodiment of the present application comprises a visual feature extraction module 100 configured to acquire a street view image of a target urban street tree, and input the street view image into an encoder of a pre-constructed depth estimation base model to obtain corresponding multi-scale visual features, and acquire a target visual feature corresponding to the multi-scale visual features by using a preset visual feature extraction network layer and a residual connection layer; a depth feature extraction module 200 configured to input the multi-scale visual features into a decoder of the depth estimation base model to extract implicit features corresponding to the street view image, and acquire a target depth feature corresponding to the implicit features by using a preset depth feature extraction network layer and a residual connection layer; and a query module 300 configured to generate a tree boundary box corresponding to each street tree in the street view image by using a preset tree target detection algorithm, calculate a boundary box embedding of a tree to be queried and a query vector corresponding to the boundary box embedding according to each tree boundary box and the target visual feature, and input the boundary box embedding, the query vector, the target visual feature, and the target depth feature into a pre-constructed three-dimensional shape regression model to output three-dimensional shape parameters of the tree to be queried in the street view image. The method for efficiently and accurately calculating the three-dimensional shape parameters of the trees from the street view images has important scientific significance and application value for promoting the urban greening management and ecological research.

[0146] Figure 7A structural schematic diagram of an electronic device is provided for an embodiment of the present application. The electronic device can include

[0147] The memory 701, the processor 702, and a computer program stored in the memory 701 and executable on the processor 702.

[0148] The processor 702 implements the street view image-based city street tree three-dimensional form parameter calculation method provided in the above embodiments when executing the program.

[0149] Further, the electronic device further includes

[0150] The communication interface 703 is used for communication between the memory 701 and the processor 702.

[0151] The memory 701 is used to store a computer program executable on the processor 702.

[0152] The memory 701 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0153] If the memory 701, the processor 702, and the communication interface 703 are independently implemented, the communication interface 703, the memory 701, and the processor 702 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0154] Optionally, in a specific implementation, if the memory 701, the processor 702, and the communication interface 703 are integrated on a chip, the memory 701, the processor 702, and the communication interface 703 can complete communication between each other through an internal interface.

[0155] The processor 702 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to perform the operations of the embodiments of the application.

[0156] The embodiments of the application further provide a computer readable storage medium, which has stored thereon a computer program, and the computer program is executed by a processor to implement the method for calculating the three-dimensional shape parameters of urban street trees based on street view images.

[0157] The embodiments of the application further provide a computer program product, which comprises a computer program, and the computer program is executed to implement the method for calculating the three-dimensional shape parameters of urban street trees based on street view images.

[0158] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or N embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0159] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0160] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing one or N executable steps, code segments, or portions of a computer program, and the scope of the preferred embodiments of the present application includes additional implementation in which the functions described are performed in different orders, including substantially simultaneously, or in reverse order, or in an order that is different from the order shown or discussed, and the like, as will be understood by those skilled in the art.

[0161] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or a combination of the above. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a propagation medium. The computer readable signal medium can include, but is not limited to, a computer readable medium that facilitates transfer of the program from one place to another. A specific example of a computer readable medium is a non-transitory computer-readable storage medium. A specific example of a computer readable signal medium is a source or destination of the computer readable medium. Another specific example of a computer readable signal medium is a computer readable signal travelling through space. Thus, a computer readable medium can take many forms of hardware to carry out the program for use by or in connection with the instruction execution system, apparatus or device.

[0162] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented with any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.

[0163] Those of skill in the art would understand that the steps of the methods carried out above can be carried out by program instructions executed by relevant hardware, and the program can be stored in a computer readable storage medium, and when the program is executed, it includes one or a combination of the steps of the method embodiments.

[0164] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0165] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for calculating the three-dimensional morphological parameters of urban street trees based on street view images, characterized in that, Includes the following steps: Obtain street view images of roadside trees in the target city and input the street view images into the encoder of a pre-built depth estimation base model to obtain the corresponding multi-scale visual features. Then, obtain the target visual features corresponding to the multi-scale visual features through a preset visual feature extraction network layer and residual connection layer. The multi-scale visual features are input into the decoder of the depth estimation base model to extract the implicit features corresponding to the street scene image, and the target depth features corresponding to the implicit features are obtained by using the preset depth feature extraction network layer and the residual connection layer. A tree bounding box corresponding to each roadside tree in the street view image is generated using a preset tree target detection algorithm. The bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding are calculated based on each tree bounding box and the target visual features. The bounding box embedding, the query vector, the target visual features and the target depth features are input into a pre-constructed three-dimensional morphological regression model to output the three-dimensional morphological parameters of the tree to be queried in the street view image. The step of inputting the street view image into the encoder of a pre-constructed depth estimation base model to obtain corresponding multi-scale visual features, and obtaining the target visual features corresponding to the multi-scale visual features through a preset visual feature extraction network layer and residual connection layer, includes: The depth estimation basic model is constructed by the decoder, the encoder, the visual feature extraction network layer, the depth feature extraction network layer, the residual connection layer, and the preset feedforward network. The depth estimation base model is trained, and the trained depth estimation base model is used to divide the street scene image into multiple sub-image blocks. Multi-scale visual features corresponding to the multiple sub-image blocks are extracted through a preset self-attention mechanism. The position embedding of each sub-image block in the plurality of sub-image blocks is calculated using a preset position encoding strategy, and the position embedding and the multi-scale visual features are input into the visual feature extraction network layer to obtain the visual enhancement features corresponding to the multi-scale visual features; The visual enhancement features are updated through the feedforward network and the residual connection layer to obtain the target visual features; The mathematical expression for the location encoding strategy is: in, Indicates the first i Embedded vectors at each position The 2nd t One element; and Represents the position embedding vector The two consecutive elements in the expression; C is the channel dimension; sin() and cos() represent the sine and cosine functions, respectively.

2. The method according to claim 1, characterized in that, The step of inputting the multi-scale visual features into the decoder of the depth estimation base model to extract the implicit features corresponding to the street view image, and using a preset depth feature extraction network layer and the residual connection layer to obtain the target depth features corresponding to the implicit features, includes: The decoder is fine-tuned using the pre-processed optimized dataset after data preprocessing. The implicit features of the street view image are extracted by the fine-tuned decoder, and the depth enhancement features corresponding to the implicit features are extracted by the depth feature extraction network layer. The depth enhancement features are updated using the residual connection layer to generate the target depth features.

3. The method according to claim 2, characterized in that, The method involves generating tree bounding boxes for each roadside tree in the street view image using a preset tree target detection algorithm, calculating the bounding box embedding of the query tree and the corresponding query vector based on each tree bounding box and the target visual features, and inputting the bounding box embedding, the query vector, the target visual features, and the target depth features into a pre-constructed 3D morphological regression model to output the 3D morphological parameters of the query tree in the street view image, including: The tree target detection algorithm generates tree bounding boxes corresponding to each roadside tree in the street view image, and calculates the bounding box embedding of the query tree and the query vector corresponding to the bounding box embedding based on the tree bounding boxes and the target visual features. The three-dimensional morphological regression model is constructed based on a preset cross-attention layer, self-attention layer, and multilayer perceptron. The target visual features, the target depth features, the bounding box corresponding to the tree to be queried, and the query vector are input into the three-dimensional morphological regression model, so as to use the multilayer perceptron and the preset tree mask regression and distance regression to guide the model to calculate the three-dimensional morphological parameters of the tree to be queried.

4. A device for calculating the three-dimensional morphological parameters of urban street trees based on street view images, characterized in that, include: The visual feature extraction module is used to acquire street view images of roadside trees in the target city and input the street view images into the encoder of a pre-built depth estimation base model to obtain the corresponding multi-scale visual features. The target visual features corresponding to the multi-scale visual features are obtained through a preset visual feature extraction network layer and a residual connection layer. The depth feature extraction module is used to input the multi-scale visual features into the decoder of the depth estimation base model to extract the implicit features corresponding to the street scene image, and use the preset depth feature extraction network layer and the residual connection layer to obtain the target depth features corresponding to the implicit features. The query module is used to generate tree bounding boxes corresponding to each roadside tree in the street view image using a preset tree target detection algorithm, and calculate the bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding based on each tree bounding box and the target visual features. The bounding box embedding, the query vector, the target visual features and the target depth features are input into a pre-constructed three-dimensional morphological regression model to output the three-dimensional morphological parameters of the tree to be queried in the street view image. The visual feature extraction module includes: The first modeling unit is used to construct the depth estimation basic model through the decoder, the encoder, the visual feature extraction network layer, the depth feature extraction network layer, the residual connection layer and the preset feedforward network; The training unit is used to train the depth estimation base model, and to divide the street scene image into multiple sub-image blocks using the trained depth estimation base model, and to extract the multi-scale visual features corresponding to the multiple sub-image blocks through a preset self-attention mechanism. The first computing unit is used to calculate the position embedding of each sub-image block in the plurality of sub-image blocks using a preset position encoding strategy, and input the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain the visual enhancement features corresponding to the multi-scale visual features; The first update unit is used to update the visual enhancement features through the feedforward network and the residual connection layer to obtain the target visual features; The mathematical expression for the location encoding strategy is: in, Indicates the first i Embedded vectors at each position The 2nd t One element; and Represents the position embedding vector The two consecutive elements in the expression; C is the channel dimension; sin() and cos() represent the sine and cosine functions, respectively.

5. An electronic device, characterized in that, include: The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for calculating the three-dimensional morphological parameters of urban street trees based on street view images as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method for calculating the three-dimensional morphological parameters of urban street trees based on street view images as described in any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the method for calculating the three-dimensional morphological parameters of urban street trees based on street view images as described in any one of claims 1-3.

Images