Urban street tree three-dimensional morphological parameter calculation method and device based on street view image

Abstract

The invention relates to an urban street tree three-dimensional morphological parameter calculation method and device based on a streetscape image, and the method comprises the steps: extracting robust visual features from the streetscape image through a depth estimation basic model and a visual feature extraction network layer; the robust depth features are obtained from the streetscape image by means of the depth estimation basic model and the depth feature extraction network layer, fixed prior and manual interpretation are not needed, and the accuracy of three-dimensional form calculation is improved; a three-dimensional form regression model is constructed, interaction of vision and depth information is realized through an attention layer, meanwhile, tree mask regression and distance regression auxiliary tasks are introduced, the visual shape and the spatial position of the tree are focused, and an estimation result of form parameters of the tree is obtained. Therefore, by constructing the end-to-end deep learning network model, the three-dimensional morphological parameters of the street tree can be directly predicted when the street view image is given, the model prediction precision is greatly improved, and the requirements of rapid monitoring and management are met.

Description

Technical Field

[0001] This application relates to the technical fields of computer vision and street view image applications, and particularly relates to a method and device for calculating three-dimensional morphological parameters of urban street trees based on street view images. Background Art

[0002] Street trees refer to tree species planted on both sides of roads and in the median strip, providing shade for vehicles and pedestrians and constituting street views, and playing a key role in maintaining the carbon-oxygen balance, regulating the urban climate, improving environmental quality, and protecting biodiversity; systematically and accurately measuring the morphology of street trees is of great significance for urban greening management and urban carbon storage estimation, etc.

[0003] Traditional measurement methods rely on professionals to conduct on-site surveys using tape measures and altimeters. Although the accuracy is relatively high, they require a large amount of manpower and time investment, and the measurement efficiency is low; on the other hand, airborne or vehicle-mounted lidar sensors can quickly scan and capture the three-dimensional structure information of street trees, but their high data acquisition cost limits the feasibility of large-scale applications and is only applicable to small-scale plot studies.

[0004] In recent years, street view images, as a low-cost and widely covered observation means, have gradually become an important data source for urban research. Many researchers have begun to use street view images to evaluate some physical attributes of street trees, such as species and location. However, existing methods for measuring tree morphology based on street view images still have many limitations; firstly, many methods rely on fixed scale priors, for example, assuming that the height and angle of the camera are known, which limits their applicability and generalization ability in different scenarios; secondly, these methods usually only rely on visual cues for morphology measurement and are easily affected by complex urban environments, thus reducing the measurement accuracy; in addition, existing image processing methods are inefficient in processing large-scale data and are difficult to meet the requirements of rapid monitoring and management.

[0005] In summary, the existing technology overly relies on fixed scale priors or visual cues for morphology measurement, greatly affecting the measurement accuracy of three-dimensional morphological parameters such as the height and diameter at breast height of urban street trees, and it is difficult to meet the requirements of rapid monitoring and management, and there is an urgent need to solve this problem. Summary of the Invention

[0006] This application provides a method and device for calculating three-dimensional morphological parameters of urban street trees based on street view images to solve problems such as the existing technology overly relying on fixed scale priors or visual cues for morphology measurement, greatly affecting the measurement accuracy of three-dimensional morphological parameters such as the height and diameter at breast height of urban street trees, and being difficult to meet the requirements of rapid monitoring and management.

[0007] An embodiment of the first aspect of the present application provides a method for calculating three-dimensional morphological parameters of urban street trees based on street view images, including the following steps: obtaining a street view image of a target urban street tree, and inputting the street view image into an encoder of a pre-constructed depth estimation basic model to obtain corresponding multi-scale visual features, and obtaining target visual features corresponding to the multi-scale visual features through a preset visual feature extraction network layer and a residual connection layer; inputting the multi-scale visual features into a decoder of the depth estimation basic model to extract implicit features corresponding to the street view image, and obtaining target depth features corresponding to the implicit features by using a preset depth feature extraction network layer and the residual connection layer; using a preset tree target detection algorithm to generate a tree bounding box corresponding to each street tree in the street view image, and calculating a bounding box embedding of a tree to be queried and a query vector corresponding to the bounding box embedding according to each tree bounding box and the target visual features, and inputting the bounding box embedding, the query vector, the target visual features, and the target depth features into a pre-constructed three-dimensional morphology regression model to output three-dimensional morphological parameters of the tree to be queried in the street view image.

[0008] Optionally, in an embodiment of the present application, the step of inputting the street view image into an encoder of a pre-constructed depth estimation basic model to obtain corresponding multi-scale visual features, and obtaining target visual features corresponding to the multi-scale visual features through a preset visual feature extraction network layer and a residual connection layer includes: constructing the depth estimation basic model through the decoder, the encoder, the visual feature extraction network layer, the depth feature extraction network layer, the residual connection layer, and a preset feed-forward network; training the depth estimation basic model, and using the trained depth estimation basic model to divide the street view image into multiple sub-image blocks, and extracting multi-scale visual features corresponding to the multiple sub-image blocks through a preset self-attention mechanism; calculating a position embedding of each sub-image block in the multiple sub-image blocks by using a preset position encoding strategy, and inputting the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain a visually enhanced feature corresponding to the multi-scale visual features; updating the visually enhanced feature through the feed-forward network and the residual connection layer to obtain the target visual features.

[0009] Optionally, in an embodiment of the present application, inputting the multi-scale visual features into the decoder of the depth estimation basic model to extract the implicit features corresponding to the street view image, and using a preset depth feature extraction network layer and the residual connection layer to obtain the target depth features corresponding to the implicit features includes: performing data preprocessing on a preset optimized dataset, and fine-tuning the decoder using the preprocessed optimized dataset; extracting the implicit features of the street view image through the fine-tuned decoder, and extracting the depth enhancement features corresponding to the implicit features through the depth feature extraction network layer; using the residual connection layer to update the depth enhancement features to generate the target depth features.

[0010] Optionally, in an embodiment of the present application, using a preset tree target detection algorithm to generate a tree bounding box corresponding to each street tree in the street view image, calculating a bounding box embedding of the tree to be queried and a query vector corresponding to the bounding box embedding according to each tree bounding box and the target visual features, and inputting the bounding box embedding, the query vector, the target visual features, and the target depth features into a pre-constructed three-dimensional shape regression model to output the three-dimensional shape parameters of the tree to be queried in the street view image, includes: generating the tree bounding box corresponding to each street tree in the street view image through the tree target detection algorithm, and calculating the bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding according to the tree bounding box and the target visual features; constructing the three-dimensional shape regression model based on a preset cross-attention layer, self-attention layer, and multi-layer perceptron; inputting the target visual features, the target depth features, the bounding box embedding corresponding to the tree to be queried, and the query vector into the three-dimensional shape regression model to use the multi-layer perceptron and a preset tree mask regression and distance regression guidance model to perform regression calculation on the three-dimensional shape parameters of the tree to be queried.

[0011] Optionally, in an embodiment of the present application, the mathematical expression of the position encoding strategy is:

[0012]

[0013] where pe i,2t represents the 2t-th element of the i-th position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() represent the sine function and cosine function respectively.

[0014] In the second aspect of the embodiments of the present application, a device for calculating three-dimensional morphological parameters of urban roadside trees based on street view images is provided, including: a visual feature extraction module, configured to obtain a street view image of a target urban roadside tree, and input the street view image into an encoder of a pre-constructed depth estimation basic model to obtain corresponding multi-scale visual features, and obtain target visual features corresponding to the multi-scale visual features through a preset visual feature extraction network layer and a residual connection layer; a depth feature extraction module, configured to input the multi-scale visual features into a decoder of the depth estimation basic model to extract implicit features corresponding to the street view image, and obtain target depth features corresponding to the implicit features by using a preset depth feature extraction network layer and the residual connection layer; a query module, configured to generate a tree bounding box corresponding to each roadside tree in the street view image by using a preset tree target detection algorithm, calculate a bounding box embedding of a tree to be queried and a query vector corresponding to the bounding box embedding according to each tree bounding box and the target visual features, and input the bounding box embedding, the query vector, the target visual features, and the target depth features into a pre-constructed three-dimensional morphology regression model to output three-dimensional morphological parameters of the tree to be queried in the street view image.

[0015] Optionally, in an embodiment of the present application, the visual feature extraction module includes: a first modeling unit, configured to construct the depth estimation basic model through the decoder, the encoder, the visual feature extraction network layer, the depth feature extraction network layer, the residual connection layer, and a preset feed-forward network; a training unit, configured to train the depth estimation basic model, divide the street view image into multiple sub-image blocks by using the trained depth estimation basic model, and extract multi-scale visual features corresponding to the multiple sub-image blocks through a preset self-attention mechanism; a first calculation unit, configured to calculate a position embedding of each sub-image block in the multiple sub-image blocks by using a preset position encoding strategy, and input the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain a visually enhanced feature corresponding to the multi-scale visual features; a first update unit, configured to update the visually enhanced feature through the feed-forward network and the residual connection layer to obtain the target visual features.

[0016] Optionally, in an embodiment of the present application, the depth feature extraction module includes: a fine-tuning unit, configured to perform data preprocessing on a preset optimized data set, and fine-tune the decoder by using the optimized data set after data preprocessing; an extraction unit, configured to extract implicit features of the street view image through the fine-tuned decoder, and extract a depth-enhanced feature corresponding to the implicit feature through the depth feature extraction network layer; a second update unit, configured to update the depth-enhanced feature by using the residual connection layer to generate the target depth features.

[0017] Optionally, in an embodiment of the present application, the query module includes: a generation unit configured to generate a tree bounding box corresponding to each street tree in the street view image through the tree target detection algorithm, and calculate the bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding according to the tree bounding box and the target visual feature; a second modeling unit configured to construct the three-dimensional shape regression model based on a preset cross-attention layer, self-attention layer, and multi-layer perceptron; and a second calculation unit configured to input the target visual feature, the target depth feature, the bounding box embedding corresponding to the tree to be queried, and the query vector into the three-dimensional shape regression model, so as to use the multi-layer perceptron and a preset tree mask regression and distance regression guidance model to perform regression calculation on the three-dimensional shape parameters of the tree to be queried.

[0018] Optionally, in an embodiment of the present application, the mathematical expression of the position encoding strategy is:

[0019]

[0020] where pe i,2t represents the 2t-th element of the i-th position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() represent the sine function and cosine function respectively.

[0021] An embodiment of the third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the method for calculating three-dimensional shape parameters of urban street trees based on street view images as described in the above embodiments.

[0022] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the program is executed by a processor, it implements the method for calculating three-dimensional shape parameters of urban street trees based on street view images as described above.

[0023] An embodiment of the fifth aspect of the present application provides a computer program product, including a computer program, and the computer program is executed to implement the method for calculating three-dimensional shape parameters of urban street trees based on street view images as described above.

[0024] Therefore, the embodiments of the present application have the following beneficial effects:

[0025] Embodiments of the present application can obtain street view images of street trees in a target city, input the street view images into the encoder of a pre-constructed depth estimation basic model to obtain corresponding multi-scale visual features, and obtain target visual features corresponding to the multi-scale visual features through a preset visual feature extraction network layer and residual connection layer; input the multi-scale visual features into the decoder of the depth estimation basic model to extract implicit features corresponding to the street view images, and use a preset depth feature extraction network layer and residual connection layer to obtain target depth features corresponding to the implicit features; use a preset tree target detection algorithm to generate tree bounding boxes corresponding to each street tree in the street view image, calculate the bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding according to each tree bounding box and the target visual features, and input the bounding box embedding, query vector, target visual features and target depth features into a pre-constructed three-dimensional shape regression model to output the three-dimensional shape parameters of the tree to be queried in the street view image. The present application can efficiently and accurately calculate the three-dimensional shape parameters of trees from street view images, which has important scientific significance and application value for promoting urban greening management and ecological research. Thus, it solves the problems in the prior art that rely too much on fixed scale priors or visual cues for morphological measurement, greatly affecting the measurement accuracy of three-dimensional shape parameters such as the height and diameter at breast height of urban street trees, and it is difficult to meet the requirements of rapid monitoring and management.

[0026] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. Brief Description of the Drawings

[0027] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0028] Figure 1 is a flowchart of a method for calculating three-dimensional shape parameters of urban street trees based on street view images according to an embodiment of the present application;

[0029] Figure 2 is a schematic diagram of the logical architecture of a method for calculating three-dimensional shape parameters of urban street trees based on street view images provided by an embodiment of the present application;

[0030] Figure 3 is a schematic diagram of a model architecture provided by an embodiment of the present application;

[0031] Figure 3 in which (a) is a schematic diagram of the architecture of a visual feature extraction model provided by an embodiment of the present application;

[0032] Figure 3Among them, (b) is a schematic diagram of a deep feature extraction model architecture provided by an embodiment of the present application;

[0033] Figure 3 Among them, (c) is a schematic diagram of a morphological parameter regression calculation model architecture provided by an embodiment of the present application;

[0034] Figure 4 It is a schematic diagram of the composition of a data set provided by an embodiment of the present application;

[0035] Figure 5 It is a schematic diagram of a prediction result provided by an embodiment of the present application;

[0036] Figure 6 It is an example diagram of a three-dimensional morphological parameter calculation device for urban roadside trees based on street view images according to an embodiment of the present application;

[0037] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application.

[0038] Among them, 10 - three-dimensional morphological parameter calculation device for urban roadside trees based on street view images; 100 - visual feature extraction module, 200 - deep feature extraction module, 300 - query module; 701 - memory, 702 - processor, 703 - communication interface. Detailed implementation manners

[0039] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application and should not be construed as a limitation to the present application.

[0040] The following describes a method and apparatus for calculating three-dimensional morphological parameters of urban roadside trees based on street view images according to embodiments of the present application. In view of the problems mentioned in the above background art, the present application provides a method for calculating three-dimensional morphological parameters of urban roadside trees based on street view images. In this method, by obtaining a street view image of a target urban roadside tree and inputting the street view image into an encoder of a pre-constructed depth estimation basic model, corresponding multi-scale visual features are obtained, and target visual features corresponding to the multi-scale visual features are obtained through a preset visual feature extraction network layer and a residual connection layer; the multi-scale visual features are input into a decoder of the depth estimation basic model to extract implicit features corresponding to the street view image, and target depth features corresponding to the implicit features are obtained by using a preset depth feature extraction network layer and a residual connection layer; a tree target detection algorithm is used to generate a tree bounding box corresponding to each roadside tree in the street view image, and a bounding box embedding of the tree to be queried and a query vector corresponding to the bounding box embedding are calculated according to each tree bounding box and the target visual features, and the bounding box embedding, the query vector, the target visual features, and the target depth features are input into a pre-constructed three-dimensional morphology regression model to output three-dimensional morphological parameters of the tree to be queried in the street view image. The method of the present application can efficiently and accurately calculate the three-dimensional morphological parameters of trees from street view images, which has important scientific significance and application value for promoting urban greening management and ecological research. Thus, the problems in the prior art that rely too much on fixed scale priors or visual cues for morphological measurement, greatly affect the measurement accuracy of three-dimensional morphological parameters such as the height and diameter at breast height of urban roadside trees, and are difficult to meet the requirements of rapid monitoring and management are solved.

[0041] Specifically, Figure 1 FIG. is a flowchart of a method for calculating three-dimensional morphological parameters of urban roadside trees based on street view images provided by an embodiment of the present application.

[0042] As Figure 1 shown, the method for calculating three-dimensional morphological parameters of urban roadside trees based on street view images includes the following steps:

[0043] In step S101, a street view image of a target urban roadside tree is obtained, and the street view image is input into an encoder of a pre-constructed depth estimation basic model to obtain corresponding multi-scale visual features, and target visual features corresponding to the multi-scale visual features are obtained through a preset visual feature extraction network layer and a residual connection layer.

[0044] In the embodiments of the present application, first, a street view image can be input into the encoder of a pre-trained depth estimation base model, and multi-scale visual features of the street view image can be extracted through a self-attention mechanism. At the same time, the embodiments of the present application introduce a learnable visual Transformer module to further enhance the visual features, and use residual connections to update the input features, so as to obtain robust visual features (i.e., target visual features).

[0045] Optionally, in an embodiment of the present application, a street view image is input into the encoder of a pre-constructed depth estimation base model to obtain corresponding multi-scale visual features, and target visual features corresponding to the multi-scale visual features are obtained through a preset visual feature extraction network layer and a residual connection layer, including: constructing a depth estimation base model through a decoder, an encoder, a visual feature extraction network layer, a depth feature extraction network layer, a residual connection layer, and a preset feed-forward network; training the depth estimation base model, and using the trained depth estimation base model to divide the street view image into multiple sub-image blocks, and extracting multi-scale visual features corresponding to the multiple sub-image blocks through a preset self-attention mechanism; calculating the position embedding of each sub-image block in the multiple sub-image blocks by using a preset position encoding strategy, and inputting the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain visual enhancement features corresponding to the multi-scale visual features; updating the visual enhancement features through a feed-forward network and a residual connection layer to obtain target visual features.

[0046] In the actual execution process, the embodiments of the present application can first input a street view image into the encoder of a pre-trained depth estimation base model (Depth Anything). This model extracts multi-level visual features of the street view image through a self-attention mechanism, further enhances the visual features through a learnable visual Transformer module, and uses residual connections to update the input features to obtain robust visual features (i.e., target visual features) F v , such as Figure 2 shown.

[0047] Those skilled in the art should understand that since the visual base model (Depth Anything) is trained on a large amount of image data exceeding 60 million and has strong generalization ability, it can help the network utilize the powerful data prior knowledge in the base model to replace the traditional prior knowledge that relies on manual setting. The embodiments of the present application can select the visual base model (Depth Anything) as the backbone network, such as Figure 3As shown in (a) therein; the Depth Anything encoder is a Vision Transformer (ViT) model, which can first divide the input street view image into L image patches (i.e., multiple sub-image patches) with a size of 14×14; secondly, extract multi-level visual features (i.e., multi-scale visual features) through a series of stacked attention blocks wherein represents the i-th level feature, H and W respectively represent the height and width of the image, and C represents the channel dimension.

[0048] Optionally, in an embodiment of the present application, the mathematical expression of the position encoding strategy is:

[0049]

[0050] wherein, pe i,2t represents the 2t-th element of the i-th position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() respectively represent the sine function and the cosine function.

[0051] For the multi-level feature F b , the higher the level, the deeper the obtained feature. In the specific implementation process, in order to exclude the influence of low-level features, the embodiment of the present application only uses the deepest feature as the input to the visual transformer module to extract visual cues; the feature map is flattened into feature tokens wherein represents the number of tokens after flattening, C represents the channel dimension; while flattening, the position embedding is calculated for each token The mathematical expression of the position encoding strategy is as follows:

[0052]

[0053] wherein, pe i,2t represents the 2t-th element of the i-th position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() respectively represent the sine function and the cosine function.

[0054] Secondly, the embodiment of the present application can use the feature tokens and the corresponding position embedding pe are input into a self-attention layer, and the calculation formula of the self-attention layer is expressed as,

[0055] A q ,A k =Linear(F)+PE,A v =Linear(F)

[0056]

[0057] where A q 、A k 、A v represent the Query matrix, Key matrix, and Value matrix respectively; F is the input feature map; PE is the input position embedding; C is the channel dimension; Linear() is a linear transformation function, which can be expressed as WF + b; Softmax() is used to normalize the weights so that the sum of all weights is 1.

[0058] Subsequently, the feature SA(F, PE) extracted by the self-attention layer is input into a feed-forward network (FFN) composed of two linear layers and an activation function. The feed-forward network layer can be expressed by the following formula:

[0059] FFN(F)=Linear(LeakyReLU(Linear(F)))

[0060] where F is the input feature map; Linear() is a linear transformation function; LeakyReLU() is an activation function used to introduce non-linear features, and the definition of LeakyReLU() is shown in the following formula:

[0061]

[0062] Finally, the embodiments of the present application can update the input features using residual connections to obtain the visual feature F v .

[0063] In summary, the mathematical expression for obtaining robust visual features by inputting street view images into a pre-trained depth estimation base model and combining self-attention mechanisms, learnable visual transformer modules, and residual connections in the embodiments of the present application is:

[0064]

[0065] It can be understood that in the embodiments of the present application, first, a street view image is input into a pre-trained depth estimation basic model, Depth Anything. This model divides the street view image into multiple image patches through a self-attention mechanism and extracts multi-level visual features of the street view image. Secondly, the position embedding of each image patch is calculated through a position encoding formula and input into a learnable visual transformer module together with the features to further enhance the visual features. Finally, the input features are updated through a feed-forward network and a residual connection to obtain robust visual features F v , so as to be able to utilize the powerful generalization ability of the basic model while paying attention to the domain knowledge of specific tasks, laying a foundation for subsequent depth feature extraction and three-dimensional shape regression.

[0066] In step S102, the multi-scale visual features are input into the decoder of the depth estimation basic model to extract the implicit features corresponding to the street view image, and the target depth features corresponding to the implicit features are obtained by using a preset depth feature extraction network layer and a residual connection layer.

[0067] Furthermore, the embodiments of the present application also need to use the pre-trained Depth Anything model to extract the implicit features of the street view image, predict the metric depth through fine-tuning and introducing a learnable depth transformer module, and further extract robust depth cues (i.e., target depth features).

[0068] Optionally, in an embodiment of the present application, inputting the multi-scale visual features into the decoder of the depth estimation basic model to extract the implicit features corresponding to the street view image, and obtaining the target depth features corresponding to the implicit features by using a preset depth feature extraction network layer and a residual connection layer includes: performing data preprocessing on a preset optimized data set, and fine-tuning the decoder by using the preprocessed optimized data set; extracting the implicit features of the street view image through the fine-tuned decoder, and extracting the depth enhanced features corresponding to the implicit features through the depth feature extraction network layer; using the residual connection layer to update the depth enhanced features to generate the target depth features.

[0069] It should be noted that in the embodiments of the present application, first, the WHU-RSTree data set can be used to fine-tune the Depth Anything decoder; secondly, the Depth Anything decoder is used to extract the implicit features of the street view image; then, a learnable depth transformer module is used to further enhance the extracted depth features; finally, the input features are updated by using a residual connection to obtain robust depth features (i.e., target depth features) F d .

[0070] Those skilled in the art should understand that since most existing depth estimation models are trained in the disparity space, they can only capture the relative distance relationship between objects and lack real scale information.

[0071] To achieve more accurate metric depth prediction, the embodiments of the present application can first fine-tune the Depth Anything decoder using the WHU-RSTree dataset. The WHU-RSTree dataset consists of point cloud data of about 68 kilometers in Nanjing, China, and 12,447 panoramic images, containing annotations of more than 20,000 trees, including instance segmentation results, tree species, and morphological parameters.

[0072] In the fine-tuning stage, the embodiments of the present application can first crop the panoramic images in the WHU-RSTree dataset using the spherical projection method. Specifically, since street trees are generally located on both sides of the road, the horizontal angles of 90° and 270° are selected as anchor points, the image size is cropped to 1024×1024, the horizontal and vertical fields of view (FoV) are set to 90°, and the images that do not contain valid street tree annotations are deleted, thus obtaining 11,011 street view images; secondly, a rough depth map is constructed by projection based on the point cloud data in the WHU-RSTree dataset, and the specific process is as follows:

[0073] 1. Project the point cloud within 95m from the camera center onto the image and calculate the Z value as the depth value. Considering that the discrete nature of the point cloud may lead to incorrect depth relationships, in order to improve the quality of the depth map;

[0074] 2. Remove the outliers, and use multi-scale dilated convolution to complete the sparse depth into a dense depth map according to the existing method;

[0075] 3. Calculate the difference between the dense depth value and the original sparse depth value. Among them, the smaller the difference, the higher the probability of becoming a foreground point. By setting a threshold for this difference, filter the depth points and obtain the final depth map.

[0076] Furthermore, the embodiments of the present application can use the existing method to identify the sky area in the street view image and force the depth of these areas to exceed a predefined maximum depth (generally set to 80 meters) to ensure that the depth values in the long-distance area will not be underestimated, thereby improving the overall prediction accuracy; finally, the embodiments of the present application can introduce the limiting conditions of surface discovery and confidence level score to improve the accuracy of the metric depth D. It can be understood that surface discovery can help the model understand the directionality of the object surface, while the confidence level score can indicate the confidence level of the model for the depth prediction of a specific area, and these constraints help to improve the quality of the prediction results.

[0077] As shown in Figure 3 (b) of [], the Depth Anything decoder is based on the DPT (Dense Prediction Transformer) architecture, which receives features F at different levels from the Depth Anything encoder b as input, and learns multi-scale and high-level abstract representations, i.e., implicit features f of the street view image, through a series of complex convolutional layers and attention mechanisms d .

[0078] After that, the embodiments of the present application can flatten the feature map into feature tokens where represents the number of tokens after flattening, and C represents the channel dimension; while flattening, positional embeddings are calculated for each token The calculation formula is as follows:

[0079]

[0080] where pe i is the positional embedding vector of the i-th feature token; pe i,2t represents the 2t-th element of the i-th positional embedding vector pe i ; pe i,2t and pe i,2t+1 are two consecutive elements in the positional embedding vector pe i ; C is the channel dimension; sin() and cos() represent the sine function and cosine function respectively.

[0081] Furthermore, the embodiments of the present application can input the feature tokens f d and the corresponding positional embeddings pe into a self-attention layer, and the calculation formula of this self-attention layer is expressed as:

[0082] A q , A k = Linear(F) + PE, A v = Linear(F)

[0083]

[0084] where A q , A k , A vrepresent the Query matrix, Key matrix, and Value matrix respectively; F is the input feature map; PE is the input position embedding; C is the channel dimension; Linear() is a linear transformation function, which can be expressed as WF + b; Softmax() is used to normalize the weights so that the sum of all weights is 1.

[0085] Subsequently, the feature SA(F, PE) obtained by the self-attention layer can be input into a feed-forward network composed of two linear layers and an activation function. The feed-forward network layer can be expressed by the following formula:

[0086] FFN(F) = Linear(LeakyReLU(Linear(F)))

[0087] where F is the input feature map; Linear() is the linear transformation function; LeakyReLU() is the activation function used to introduce non-linear features, and the definition of LeakyReLU() is as follows:

[0088]

[0089] Finally, the embodiment of the present application can update the input feature f f , to obtain F d , specifically, the embodiment of the present application first needs to downsample the metric depth D using the average pooling method to match the resolution of the feature f d ; then calculate the depth embedding DE corresponding to each feature token, and the calculation formula is as follows:

[0090]

[0091] where de i is the depth embedding vector of the i-th feature token; de i,2t represents the 2t-th element of the i-th depth embedding vector de i ; de i,2t and de i,2t+1 are two consecutive elements in the depth embedding vector de i ; d i is the average metric depth of each feature token; C is the channel dimension; sin() and cos() represent the sine function and cosine function respectively.

[0092] It can be understood that the above sine encoding can effectively capture the differences between depth values and improve the network's ability to represent spatial information.

[0093] In summary, the embodiments of this application use the pre-trained Depth Anything model to extract the implicit features of street view images, and predict the metric depth through fine-tuning and introducing a learnable depth transformer module. Furthermore, the mathematical expression for extracting robust depth cues is

[0094] F d = f d + FFN(SA(f d , PE)) + DE

[0095] Therefore, the embodiments of this application developed a new tree morphology dataset TMD by post-processing the data of the WHU-RSTree dataset to fill the gap in the field, so as to effectively support the morphological analysis of roadside trees and promote the development of the relevant research community.

[0096] In step S103, a tree bounding box corresponding to each street tree in the street view image is generated using a preset tree object detection algorithm, and the bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding are calculated based on each tree bounding box and the target visual features. Then, the bounding box embedding, query vector, target visual features, and target depth features are input into a pre-constructed three-dimensional morphology regression model to output the three-dimensional morphology parameters of the tree to be queried in the street view image.

[0097] Furthermore, the embodiments of this application also need to construct a three-dimensional morphology regression model, regard the tree as a query vector, stack attention layers in the regression model to promote the in-depth interaction between visual and depth information, and introduce two auxiliary regression tasks, namely tree mask regression and distance regression guidance model. One guides the model to focus on the visual shape of the tree in the image, and the other emphasizes the spatial position of the tree in space, so as to achieve efficient and accurate morphological parameter estimation.

[0098] Optionally, in an embodiment of the present application, a preset tree target detection algorithm is used to generate a tree bounding box corresponding to each street tree in the street view image, and the bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding are calculated according to each tree bounding box and the target visual feature. Then, the bounding box embedding, query vector, target visual feature, and target depth feature are input into a pre-constructed three-dimensional shape regression model to output the three-dimensional shape parameters of the tree to be queried in the street view image, including: generating a tree bounding box corresponding to each street tree in the street view image through the tree target detection algorithm, and calculating the bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding according to the tree bounding box and the target visual feature; constructing a three-dimensional shape regression model based on a preset cross-attention layer, self-attention layer, and multi-layer perceptron; inputting the target visual feature, target depth feature, bounding box embedding, and query vector corresponding to the tree to be queried into the three-dimensional shape regression model to use the multi-layer perceptron and the preset tree mask regression and distance regression guidance model to perform regression calculation on the three-dimensional shape parameters of the tree to be queried.

[0099] As Figure 3 shown in (c) of i , to fuse the visual information of the bounding box, the embodiment of the present application can first design a new bounding box prompt embedding strategy to fuse and obtain the bounding box embedding be i and the query vector q i ; then, using the bounding box embedding be i , query vector q d , depth feature F v , and visual feature F i as inputs, the cross-attention layer and self-attention layer are used to promote the model to pay more attention to the interaction of the depth information and visual information of the tree within the bounding box; finally, with the help of the multi-layer perceptron and two auxiliary regression tasks, the shape parameter p i = {dbh i} is jointly regressed and calculated.

[0100] It should be noted that the embodiment of the present application does not directly detect the street view image, but uses the bounding box containing the tree detected by the target detection method as the input, so as to benefit from the excellent performance of the target detection model trained on a large-scale dataset and efficiently obtain more robust and accurate parameter estimation results.

[0101] Specifically, taking the street view image as the input and adopting an existing target detection method, the embodiment of the present application can obtain several bounding boxes containing independent trees For each bounding box b i , its center coordinate is used as the anchor point, and the corresponding feature F v is extracted to initialize the query vector This method of obtaining query vectors based on anchor points effectively reduces the search space of the network, thus promoting faster and more accurate convergence.

[0102] Secondly, the embodiments of the present application can encode the height and width information of the bounding box to construct the bounding box embedding vector be i , and the specific process is as follows:

[0103] 1. Calculate the embeddings of the height and width of each bounding box b i using the formula. The calculation formula is as follows:

[0104]

[0105] where he i and we i are the height and width embedding vectors of the i-th bounding box respectively; he i,2t represents the 2t-th element of the height embedding vector he i of the i-th bounding box; he i,2t and he i,2t+1 are two consecutive elements in the embedding vector he i ; h i is the height of each bounding box; C is the channel dimension; sin() and cos() represent the sine function and cosine function respectively. The calculation formula for the width embedding vector is the same.

[0106] 2. Calculate the position embedding vector pe i corresponding to the query q i , and introduce three learnable parameters {E h , E w , E p}. E h , E w , E p are the identifiers of height embedding, width embedding and position embedding respectively. The introduction of identifiers helps the network learn the specific representations of different elements and improve the ability to express different types of information.

[0107] 3. Use a linear function to fuse these embeddings to obtain the bounding box embedding be i , and the mathematical expression is as follows:

[0108] be i = Linear(Concatenate((he i + E h ), (we i + E w ), (pe i + E p )))

[0109] Among them, Concatenate() represents the concatenation operation in the channel dimension, that is, different types are concatenated in the channel dimension.

[0110] After that, an embodiment of the present application can give a bounding box embedding be i and its corresponding query vector q i , and at the same time input the target visual feature F v and the target depth feature F d ; The three-dimensional shape parameter regression model MPRM can use the cross-attention layer (Cross-Attention Layer, CA) and the self-attention layer (Self-Attention Layer, SA) to achieve visual and depth information interaction. The mathematical expression of this fusion process is:

[0111] A q = Linear(Q) + BE

[0112] A k , A v = Linear(F)

[0113]

[0114] Q = Q + FFN(CA(SA(CA(Q, BE, F d ), BE), BE, F v ))

[0115] Among them, Q is the query vector; BE is the bounding box embedding vector; F is the feature map; C is the channel dimension; A q , A k , A v are the query matrix, key matrix, and value matrix obtained through the linear transformation Linear() respectively; CA and SA represent the cross-attention layer and the self-attention layer respectively; FFN represents the feed-forward neural network layer.

[0116] Through the above processing, the query vector of the embodiment of the present application can be updated and integrated with visual information and depth information; finally, based on the updated query vector Q and the original bounding box embedding BE, the three-dimensional shape parameter regression model MPRM can use a shared multi-layer perceptron to simultaneously regress the shape parameters p i = {dbh i , th i} and the distance dis i from the measured tree to the camera center; The design of the shared multi-layer perceptron can reduce the number of parameters, enabling the model to complete the prediction of multiple targets in a single forward propagation and improving the calculation efficiency; In addition, in order for the model to regress and generate the instance mask m i corresponding to each tree​​​​​​​​​, the low-level visual features are combined with the visual feature F v . After upsampling the result to the original resolution of the input image, the upsampled features are multiplied with the query vector q i through a dot product operation, and the instance mask m i corresponding to each tree is calculated through a linear layer. As Figure 4 shown, the calculation formula is as follows:

[0117]

[0118] Among them, represents the dot product operation, which is used to measure the similarity between the two, and an accurate mask of each tree instance can be obtained; calculating the distance dis i and the instance mask m i is to assist in training the network, guiding the network to focus on the visual shape of the trees in the image and their spatial positions in space, realizing efficient and accurate parameter evaluation. The evaluation prediction results are as Figure 5 shown.

[0119] In summary, the embodiment of the present application first uses a visual base model as the backbone network. This visual base model can utilize its powerful data prior knowledge to replace the traditional prior knowledge that depends on manual setting, so as to extract robust visual features; secondly, considering that depth is a robust and widely applicable data connecting 2D images and 3D worlds, which can not only provide scale priors but also complete missing spatial information, a depth base estimation model is used and a depth transformer model is constructed to extract robust depth cues; finally, a morphological parameter regression model is developed to fuse visual and depth cues and regressively calculate the morphological parameters of the trees.

[0120] Thus, the embodiment of the present application realizes an end-to-end network for quickly and accurately estimating the morphological parameters of street trees from low-cost street view images, without relying on fixed scale priors or manual interpretations. By integrating visual and depth cues with the help of a depth estimation base model, an end-to-end method for tree morphology measurement based on street view images is realized, which greatly improves the efficiency of urban tree management and vegetation carbon storage estimation.

[0121] The method for calculating three-dimensional morphological parameters of urban street trees based on street view images proposed in the embodiments of the present application extracts robust visual features from street view images by using a depth estimation basic model and a visual feature extraction network layer; obtains robust depth features from street view images with the help of the depth estimation basic model and the depth feature extraction network layer, without fixed priors and manual interpretations, improving the accuracy of three-dimensional morphological calculations; constructs a three-dimensional morphological regression model, realizes the interaction of visual and depth information through an attention layer, and at the same time introduces tree mask regression and distance regression auxiliary tasks, focusing on the visual shape and spatial position of trees to obtain the estimation results of the morphological parameters of trees. The present application can efficiently and accurately calculate the three-dimensional morphological parameters of trees from street view images, which has important scientific significance and application value for promoting urban greening management and ecological research.

[0122] Secondly, a device for calculating three-dimensional morphological parameters of urban street trees based on street view images proposed in the embodiments of the present application will be described with reference to the accompanying drawings.

[0123] Figure 6 It is a block diagram of a device for calculating three-dimensional morphological parameters of urban street trees based on street view images according to an embodiment of the present application.

[0124] As Figure 6 shown, the device 10 for calculating three-dimensional morphological parameters of urban street trees based on street view images includes: a visual feature extraction module 100, a depth feature extraction module 200, and a query module 300.

[0125] Among them, the visual feature extraction module 100 is used to obtain the street view image of the target urban street tree, and input the street view image into the encoder of the pre-constructed depth estimation basic model to obtain the corresponding multi-scale visual features, and obtain the target visual features corresponding to the multi-scale visual features through the preset visual feature extraction network layer and the residual connection layer.

[0126] The depth feature extraction module 200 is used to input the multi-scale visual features into the decoder of the depth estimation basic model to extract the implicit features corresponding to the street view image, and obtain the target depth features corresponding to the implicit features by using the preset depth feature extraction network layer and the residual connection layer.

[0127] The query module 300 is used to generate the tree bounding boxes corresponding to each street tree in the street view image by using the preset tree target detection algorithm, calculate the bounding box embedding and the query vector corresponding to the bounding box embedding of the tree to be queried according to each tree bounding box and the target visual features, and input the bounding box embedding, the query vector, the target visual features, and the target depth features into the pre-constructed three-dimensional morphological regression model to output the three-dimensional morphological parameters of the tree to be queried in the street view image.

[0128] Optionally, in an embodiment of the present application, the visual feature extraction module 100 includes: a first modeling unit, a training unit, a first calculation unit, and a first update unit.

[0129] Among them, the first modeling unit is used to construct a depth estimation basic model through a decoder, an encoder, a visual feature extraction network layer, a depth feature extraction network layer, a residual connection layer, and a preset feed-forward network.

[0130] The training unit is used to train the depth estimation basic model, and use the trained depth estimation basic model to divide the street view image into multiple sub-image blocks, and extract multi-scale visual features corresponding to the multiple sub-image blocks through a preset self-attention mechanism.

[0131] The first calculation unit is used to calculate the position embedding of each sub-image block in the multiple sub-image blocks by using a preset position encoding strategy, and input the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain visual enhancement features corresponding to the multi-scale visual features.

[0132] The first update unit is used to update the visual enhancement features through a feed-forward network and a residual connection layer to obtain target visual features.

[0133] Optionally, in an embodiment of the present application, the depth feature extraction module 200 includes: a fine-tuning unit, an extraction unit, and a second update unit.

[0134] Among them, the fine-tuning unit is used to perform data preprocessing on a preset optimized data set, and use the preprocessed optimized data set to fine-tune the decoder.

[0135] The extraction unit is used to extract implicit features of the street view image through the fine-tuned decoder, and extract depth enhancement features corresponding to the implicit features through a depth feature extraction network layer.

[0136] The second update unit is used to update the depth enhancement features by using a residual connection layer to generate target depth features.

[0137] Optionally, in an embodiment of the present application, the query module 300 includes: a generation unit, a second modeling unit, and a second calculation unit.

[0138] Among them, the generation unit is used to generate a tree bounding box corresponding to each street tree in the street view image through a tree target detection algorithm, and calculate a bounding box embedding of the tree to be queried and a query vector corresponding to the bounding box embedding according to the tree bounding box and the target visual features.

[0139] The second modeling unit is used to construct a three-dimensional shape regression model based on a preset cross-attention layer, a self-attention layer, and a multi-layer perceptron.

[0140] A second computing unit, configured to input the target visual feature, the target depth feature, the bounding box embedding corresponding to the tree to be queried, and the query vector into a three-dimensional shape regression model, so as to use a multi-layer perceptron and a preset tree mask regression and distance regression guidance model to perform regression calculation on the three-dimensional shape parameters of the tree to be queried.

[0141] Optionally, in an embodiment of the present application, the mathematical expression of the position encoding strategy is:

[0142]

[0143] where pe i,2t represents the 2t-th element of the i-th position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() respectively represent the sine function and the cosine function.

[0144] It should be noted that the foregoing explanation of the embodiment of the method for calculating the three-dimensional shape parameters of urban roadside trees based on street view images also applies to the device for calculating the three-dimensional shape parameters of urban roadside trees based on street view images in this embodiment, and will not be elaborated here.

[0145] The device for calculating the three-dimensional shape parameters of urban roadside trees based on street view images according to the embodiments of the present application includes a visual feature extraction module 100, configured to obtain a street view image of a target urban roadside tree, and input the street view image into an encoder of a pre-constructed depth estimation basic model to obtain corresponding multi-scale visual features, and obtain target visual features corresponding to the multi-scale visual features through a preset visual feature extraction network layer and a residual connection layer; a depth feature extraction module 200, configured to input the multi-scale visual features into a decoder of the depth estimation basic model to extract implicit features corresponding to the street view image, and obtain target depth features corresponding to the implicit features through a preset depth feature extraction network layer and a residual connection layer; a query module 300, configured to use a preset tree target detection algorithm to generate a tree bounding box corresponding to each roadside tree in the street view image, calculate a bounding box embedding corresponding to the tree to be queried and a query vector corresponding to the bounding box embedding according to each tree bounding box and the target visual feature, and input the bounding box embedding, the query vector, the target visual feature, and the target depth feature into a pre-constructed three-dimensional shape regression model to output the three-dimensional shape parameters of the tree to be queried in the street view image. The method of the present application can efficiently and accurately calculate the three-dimensional shape parameters of trees from street view images, which has important scientific significance and application value for promoting urban greening management and ecological research.

[0146] Figure 7Schematic diagram of the structure of the electronic device provided by the embodiment of the present application. The electronic device may include:

[0147] A memory 701, a processor 702, and a computer program stored on the memory 701 and executable on the processor 702.

[0148] When the processor 702 executes the program, it implements the method for calculating three-dimensional morphological parameters of urban roadside trees based on street view images provided in the above embodiments.

[0149] Furthermore, the electronic device further includes:

[0150] A communication interface 703 for communication between the memory 701 and the processor 702.

[0151] The memory 701 is used to store a computer program executable on the processor 702.

[0152] The memory 701 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0153] If the memory 701, the processor 702, and the communication interface 703 are implemented independently, the communication interface 703, the memory 701, and the processor 702 may be interconnected through a bus and complete communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 7 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0154] Optionally, in a specific implementation, if the memory 701, the processor 702, and the communication interface 703 are integrated on a chip, the memory 701, the processor 702, and the communication interface 703 may complete communication with each other through an internal interface.

[0155] The processor 702 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0156] The embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method for calculating three-dimensional morphological parameters of urban roadside trees based on street view images as described above is implemented.

[0157] The embodiments of the present application also provide a computer program product, including a computer program. When the computer program is executed, it is used to implement the method for calculating three-dimensional morphological parameters of urban roadside trees based on street view images as described above.

[0158] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, without conflict, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0159] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0160] Any process or method description in the flowchart or described in other ways herein may be understood as representing a module, segment, or part of code including one or N executable instructions for implementing a customized logic function or process. The scope of the preferred embodiments of the present application includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art of the embodiments of the present application.

[0161] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definitional sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in conjunction with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection part (electronic device) having one or N wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0162] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or a combination of the following techniques known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0163] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of the above-described embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0164] In addition, each functional unit in various embodiments of the present application may be integrated into a processing module, may exist physically alone for each unit, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0165] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for calculating three-dimensional morphological parameters of urban street trees based on street view images, characterized in that, It includes the following steps: Obtain the street view images of the street trees in the target city, and input the street view images into the encoder of a pre-constructed depth estimation basic model to obtain corresponding multi-scale visual features, and obtain the target visual features corresponding to the multi-scale visual features through a preset visual feature extraction network layer and a residual connection layer; Input the multi-scale visual features into the decoder of the depth estimation basic model to extract the implicit features corresponding to the street view images, and use a preset depth feature extraction network layer and the residual connection layer to obtain the target depth features corresponding to the implicit features; Use a preset tree target detection algorithm to generate the tree bounding boxes corresponding to each street tree in the street view image, calculate the bounding box embedding of the tree to be queried and the query vector corresponding to the bounding box embedding according to each tree bounding box and the target visual features, and input the bounding box embedding, the query vector, the target visual features and the target depth features into a pre-constructed three-dimensional shape regression model to output the three-dimensional shape parameters of the tree to be queried in the street view image.

2. The method according to claim 1, characterized in that, The step of inputting the street view images into the encoder of a pre-constructed depth estimation basic model to obtain corresponding multi-scale visual features, and obtaining the target visual features corresponding to the multi-scale visual features through a preset visual feature extraction network layer and a residual connection layer includes: Construct the depth estimation basic model through the decoder, the encoder, the visual feature extraction network layer, the depth feature extraction network layer, the residual connection layer and a preset feed-forward network; Train the depth estimation basic model, and use the trained depth estimation basic model to divide the street view images into multiple sub-image blocks, and extract the multi-scale visual features corresponding to the multiple sub-image blocks through a preset self-attention mechanism; Calculate the position embedding of each sub-image block in the multiple sub-image blocks by using a preset position encoding strategy, and input the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain the visually enhanced features corresponding to the multi-scale visual features; Update the visually enhanced features through the feed-forward network and the residual connection layer to obtain the target visual features.

3. The method according to claim 2, wherein The step of inputting the multi-scale visual features into the decoder of the depth estimation basic model to extract the implicit features corresponding to the street view images, and using a preset depth feature extraction network layer and the residual connection layer to obtain the target depth features corresponding to the implicit features includes: Perform data preprocessing on a preset optimized data set, and fine-tune the decoder by using the preprocessed optimized data set; Extract the implicit features of the street view images through the fine-tuned decoder, and extract the depth-enhanced features corresponding to the implicit features through the depth feature extraction network layer; Use the residual connection layer to update the depth-enhanced features to generate the target depth features.

4. The method according to claim 3, wherein Generating a tree bounding box corresponding to each street tree in the street view image by using a preset tree target detection algorithm, calculating a bounding box embedding of the tree to be queried and a query vector corresponding to the bounding box embedding according to each tree bounding box and the target visual feature, and inputting the bounding box embedding, the query vector, the target visual feature, and the target depth feature into a pre-constructed three-dimensional shape regression model to output three-dimensional shape parameters of the tree to be queried in the street view image, including: Generating a tree bounding box corresponding to each street tree in the street view image by using the tree target detection algorithm, and calculating the bounding box embedding of the tree to be queried and a query vector corresponding to the bounding box embedding according to the tree bounding box and the target visual feature; Constructing the three-dimensional shape regression model based on a preset cross-attention layer, self-attention layer, and multi-layer perceptron; Inputting the target visual feature, the target depth feature, the bounding box embedding corresponding to the tree to be queried, and the query vector into the three-dimensional shape regression model to use the multi-layer perceptron and a preset tree mask regression and distance regression guidance model to perform regression calculation on the three-dimensional shape parameters of the tree to be queried.

5. The method according to claim 2, wherein The mathematical expression of the position encoding strategy is: Among them, pe i,2t represents the 2t-th element of the i-th position embedding vector pe i ; pe i,2t and pe i,2t+1 represent two consecutive elements in the position embedding vector pe i ; C is the channel dimension; sin() and cos() represent the sine function and cosine function respectively.

6. A device for calculating three-dimensional morphological parameters of urban street trees based on street view images, characterized in that, Including: A visual feature extraction module, configured to obtain a street view image of a target urban street tree, input the street view image into an encoder of a pre-constructed depth estimation basic model to obtain corresponding multi-scale visual features, and obtain target visual features corresponding to the multi-scale visual features through a preset visual feature extraction network layer and a residual connection layer; A depth feature extraction module, configured to input the multi-scale visual features into a decoder of the depth estimation basic model to extract implicit features corresponding to the street view image, and obtain target depth features corresponding to the implicit features by using a preset depth feature extraction network layer and the residual connection layer; A query module, configured to generate a tree bounding box corresponding to each street tree in the street view image by using a preset tree target detection algorithm, calculate a bounding box embedding of the tree to be queried and a query vector corresponding to the bounding box embedding according to each tree bounding box and the target visual feature, and input the bounding box embedding, the query vector, the target visual feature, and the target depth feature into a pre-constructed three-dimensional shape regression model to output three-dimensional shape parameters of the tree to be queried in the street view image.

7. The device according to claim 6, characterized in that, The visual feature extraction module includes: A first modeling unit, configured to construct the depth estimation basic model through the decoder, the encoder, the visual feature extraction network layer, the depth feature extraction network layer, the residual connection layer, and a preset feed-forward network; A training unit, configured to train the depth estimation basic model, use the trained depth estimation basic model to divide the street view image into multiple sub-image blocks, and extract multi-scale visual features corresponding to the multiple sub-image blocks through a preset self-attention mechanism; A first computing unit, configured to calculate the position embedding of each of the plurality of sub-image patches by using a preset position encoding strategy, and input the position embedding and the multi-scale visual features into the visual feature extraction network layer to obtain visual enhancement features corresponding to the multi-scale visual features; A first updating unit, configured to update the visual enhancement features through the feed-forward network and the residual connection layer to obtain the target visual features.

8. An electronic device, characterized in that, Comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the method for calculating three-dimensional morphological parameters of urban street trees based on street view images according to any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method for calculating three-dimensional morphological parameters of urban street trees based on street view images according to any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the method for calculating three-dimensional morphological parameters of urban street trees based on street view images according to any one of claims 1-5.

Images