Urban non-fortified building reinforcement identification method and device based on historical streetscape pictures

Abstract

The invention discloses an urban non-fortification building reinforcement identification method and device based on historical streetscape pictures, and relates to the technical field of civil engineering technology and image processing, and the method comprises the steps: constructing a streetscape sequence data set containing time sequence features according to the historical streetscape pictures and streetscape metadata, the method comprises the following steps: firstly, constructing a streetscape feature extraction model, marking streetscape pictures of non-fortified buildings in different reinforcement states to obtain a streetscape training set, and training the streetscape spatial-temporal feature extraction model which is an improved FCOS network; inputting a streetscape sequence image to be recognized into the trained streetscape spatial-temporal feature extraction model, and determining a reinforcement state and a change time point of a non-fortified building in the streetscape sequence image; and in combination with geographic information data, geocoding is performed on the identified building, the spatial distribution and change trend of the reinforced building are obtained, and a building reinforcement thermodynamic diagram over the years in the region of interest is obtained. According to the invention, the efficiency of building safety management can be greatly improved, and the investment of human resources is reduced.

Description

Technical Field

[0001] The present invention relates to the fields of civil engineering technology and image processing technology, and in particular to a method and device for identifying urban non-fortified building reinforcement based on historical street scene pictures. Background Art

[0002] The reinforcement and maintenance of urban buildings are of great significance for urban planning and disaster prevention and mitigation. However, traditional building reinforcement monitoring methods usually rely on manual inspections and records, which are inefficient. With the expansion of urban scale and the increase in the number of buildings, traditional methods are becoming increasingly difficult to cope with. Compared with traditional building reinforcement monitoring methods, the urban non-fortified building reinforcement identification method based on historical street view images can efficiently monitor the reinforcement of urban non-fortified buildings on a large scale.

[0003] To identify the reinforcement of urban non-fortified buildings on a large scale, it is necessary to obtain historical street view images of the area of ​​interest and build a sample set of street view images. Then, computer technology is used to train a model that can automatically identify reinforced urban non-fortified buildings, so as to efficiently and accurately identify reinforced non-fortified buildings in street view images on a large scale. Then, through geographic information technology, according to the metadata and building footprint data in the street view, the buildings in the street view are geocoded, and the geographical location information of the reinforced buildings can be accurately located and recorded.

[0004] Prior art 1: (Wang, C., Antos, SE, Triveno, LM, et al. Automatic detection of unreinforced masonry buildings from street view images using deep learning-based image segmentation[J]. Automation in Construction, 2021,132, 103968.) proposed an automatic detection method for unfortified masonry buildings in street view images based on deep learning image segmentation. By manually annotating a street view image dataset and training a prediction model, unfortified masonry buildings in street view images can be identified. It is also proved that this method can overcome noise and occlusion problems. However, this method only focuses on building detection at a single time point and does not take into account the changes in buildings in different time periods. Therefore, it is insufficient in identifying time series changes in building reinforcement.

[0005] Prior Art 2: (Jiang, S., Lin, H., Ren, H.; et al. MDANet: A High-Resolution City Change Detection Network Based on Difference and Attention Mechanisms under Multi-Scale Feature Fusion[J]. Remote Sens, 2024, 16, 1387.) proposed a multi-scale feature fusion high-resolution urban change detection network (MDANet) based on difference and attention mechanism. This method effectively improves the accuracy of urban change detection by fusing multi-scale features and combining difference and attention mechanisms. MDANet can detect subtle changes in high-resolution images and provides high spatiotemporal resolution. However, this method mainly focuses on urban change detection and does not propose a solution for the identification of specific non-fortified building reinforcements, resulting in low efficiency and accuracy in the identification of specific non-fortified building reinforcements. Summary of the invention

[0006] In order to solve the technical problem of low efficiency and accuracy in the existing technology for identifying specific non-fortified building reinforcements, the embodiment of the present invention provides a method and device for identifying urban non-fortified building reinforcements based on historical street view pictures. The technical solution is as follows:

[0007] On the one hand, a method for identifying urban non-fortified building reinforcement based on historical street view pictures is provided, the method being implemented by an urban non-fortified building reinforcement identification device based on historical street view pictures, the method comprising:

[0008] S1. Obtain historical street view images and street view metadata of a target area, and construct a street view sequence dataset containing temporal features according to the historical street view images and street view metadata;

[0009] S2, annotating street view images of non-fortified buildings in different reinforcement states in the street view sequence data set to obtain a street view training set, and using the street view training set to train an initial street view spatiotemporal feature extraction model to obtain a trained street view spatiotemporal feature extraction model, wherein the street view spatiotemporal feature extraction model is an improved FCOS network;

[0010] S3, obtaining a street view sequence image to be identified, inputting the street view sequence image to be identified into a trained street view spatiotemporal feature extraction model, and determining the reinforcement state and change time point of the non-fortified building in the street view sequence image to be identified;

[0011] S4. Combined with geographic information data, geo-code the identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain the thermal map of building reinforcement in the area of ​​interest over the years.

[0012] On the other hand, a device for identifying urban non-fortified building reinforcement based on historical street view pictures is provided, and the device is applied to a method for identifying urban non-fortified building reinforcement based on historical street view pictures, and the device comprises:

[0013] A construction unit, configured to obtain historical street view images and street view metadata of a target area, and construct a street view sequence dataset containing temporal features according to the historical street view images and street view metadata;

[0014] A training unit is used to annotate street view images of non-fortified buildings in different reinforcement states in the street view sequence data set to obtain a street view training set, and use the street view training set to train an initial street view spatiotemporal feature extraction model to obtain a trained street view spatiotemporal feature extraction model, wherein the street view spatiotemporal feature extraction model is an improved FCOS network;

[0015] The recognition unit is used to obtain a street scene sequence image to be recognized, input the street scene sequence image to be recognized into a trained street scene spatiotemporal feature extraction model, and determine the reinforcement state and change time point of the non-fortified building in the street scene sequence image to be recognized;

[0016] The determination unit is used to combine geographic information data, geocode the identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain the thermal map of building reinforcement in the area of ​​interest over the years.

[0017] On the other hand, a device for identifying urban non-fortified building reinforcements based on historical street view images is provided, and the device for identifying urban non-fortified building reinforcements based on historical street view images comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned methods for identifying urban non-fortified building reinforcements based on historical street view images is implemented.

[0018] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for identifying reinforcement of urban non-fortified buildings based on historical street view images.

[0019] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0020] By constructing a street view sequence dataset containing temporal features and using the improved FCOS network model to extract the spatiotemporal features of the street view, the ConvNeXt-T algorithm is used to improve the nonlinear expression ability of the network and increase the network's receptive field, while reducing the number of parameters and calculations; the spatiotemporal attention module combines the information of the spatial and temporal dimensions to capture the key changes in the input feature map. This module is introduced before each head of the improved FCOS network, which can effectively improve the recognition ability of building changes in street view sequence images; furthermore, it can effectively identify the reinforcement status of non-fortified buildings in different time periods, and clearly identify the change time points in the reinforcement process. It can greatly improve the efficiency of building safety management and reduce the investment of human resources. Combining the results of street view image recognition with geographic information data, the reinforcement status of non-fortified buildings can be accurately geographically coded, and the spatial distribution of building reinforcement can be obtained, providing more accurate data support for subsequent urban planning and building safety assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a flow chart of a method for identifying urban non-fortified building reinforcement based on historical street scene pictures provided by an embodiment of the present invention;

[0023] Figure 2 It is a schematic diagram of the structure of an improved FCOS network provided by an embodiment of the present invention;

[0024] Figure 3 It is a block diagram of a device for identifying urban non-fortified building reinforcement based on historical street scene pictures provided by an embodiment of the present invention;

[0025] Figure 4 It is a structural schematic diagram of a device for identifying urban non-fortified building reinforcement based on historical street view pictures provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0027] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0028] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0029] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be written in non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.

[0030] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0031] The embodiment of the present invention provides a method for identifying urban non-fortified building reinforcement based on historical street view pictures. The method can be implemented by an urban non-fortified building reinforcement identification device based on historical street view pictures. The urban non-fortified building reinforcement identification device based on historical street view pictures can be a terminal or a server. Figure 1 The flowchart of the urban non-fortified building reinforcement identification method based on historical street view pictures is shown, and the processing flow of the method may include the following steps:

[0032] S1. Obtain historical street view images and street view metadata of the target area, and construct a street view sequence dataset containing temporal features based on the historical street view images and street view metadata.

[0033] Optionally, the step of S1 includes S11-S13:

[0034] S11. Obtain a road network data file of the target area, and select points at equal distances on the road network data file according to the density of the selected specific sampling points to determine the sampling point locations of the historical street view image.

[0035] S12. For each sampling point location, obtain a newly updated street view picture with the shortest distance from the sampling point location; wherein the newly updated street view picture refers to a street view picture with the smallest absolute value of the difference between the shooting time and the current time; each street view picture corresponds to a unique panoID value, and the panoID value is used to indicate street view metadata, and the street view metadata includes the shooting point coordinates, shooting time, vehicle head direction, true north direction and historical street view column data corresponding to the street view picture.

[0036] S13. According to the historical street view column data in the street view metadata indicated by the panoID value corresponding to each street view image, the panoID values ​​corresponding to the multiple street view images whose corresponding sampling point locations are in different historical periods are obtained; according to the panoID values ​​corresponding to the multiple street view images in different historical periods, the corresponding street view metadata are obtained; and according to the multiple street view images in different historical periods and the corresponding street view metadata, street view sequence data containing time series features are composed.

[0037] In a feasible implementation, first, obtain the road network data file (shapefile format) of the area of ​​interest from OpenStreetMap, select a specific sampling point density, such as 10m, 25m, 50m, etc., and take points at equal distances on the road network data to determine the sampling point locations of the street view images; each sampling point attribute includes latitude and longitude coordinate values;

[0038] Secondly, for each determined sampling point location, the nearest updated street view panoramic image can be obtained (that is, the Baidu Map official API can be used to obtain the latest street view image closest to the point based on the latitude and longitude coordinates). Each street view image corresponds to a unique panoID value, and based on this unique panoID value, street view metadata such as street view shooting point coordinates, street view shooting time, vehicle head direction, true north direction, historical street view column, etc. can be obtained;

[0039] Then, for the historical street view column in the street view metadata, the panoID values ​​corresponding to multiple street view images of different historical periods belonging to the sampling point can be obtained, and then the respective street view metadata can be obtained to form a street view sequence containing time series features.

[0040] Optionally, the street view sequence data containing time series features is expressed by the following formula (1):

[0041] (1)

[0042] in, Represents a collection of street view sequence data containing time series features. Indicates the longitude of the shooting point coordinates in the street view metadata. Indicates the latitude of the shooting point coordinates in the street view metadata. represents the total number of street view images in the collection, Represents the i-th street view picture with the coordinates of the shooting point.

[0043] The larger the index in the upper right corner, the closer the shooting time of the street view sequence image is to the present;

[0044] Therefore, each street view picture can be represented in a symbolic way.

[0045] S2. Annotate street view images of non-fortified buildings in different reinforcement states in the street view sequence dataset to obtain a street view training set. Use the street view training set to train the initial street view spatiotemporal feature extraction model to obtain a trained street view spatiotemporal feature extraction model. The street view spatiotemporal feature extraction model is an improved FCOS network.

[0046] Optionally, the process of training the initial street view spatiotemporal feature extraction model in S2 may include the following steps S21-S24:

[0047] S21. Annotate street view images of non-fortified buildings in different reinforcement states in the street view sequence data set to obtain street view training data.

[0048] Among them, different reinforcement states include the initial state of reinforcement, the state of reinforcement process and the final state of reinforcement;

[0049] Among them, the initial state of reinforcement refers to the state where the non-fortified building has no obvious beam and column structure on the facade, the overall structure lacks maintenance, and the surface material has aged;

[0050] The reinforcement process state refers to the state of non-fortified buildings in construction equipment accompanied by scaffolding and coverings;

[0051] The final state of reinforcement refers to the state in which the non-fortified building has an obvious beam-column structure on the facade, showing good maintenance and reinforcement effects.

[0052] In a feasible implementation, professionals annotate street view images of non-fortified buildings in three states and classify them into an initial state of reinforcement, a state in the process of reinforcement, and a final state of reinforcement to form street view training data.

[0053] S22, preprocessing the street view training data.

[0054] In a feasible implementation, the size of all street view images in the street view sequence is 4096×2048 pixels, including three RGB channels, and all images are subjected to illumination normalization processing to reduce illumination influence.

[0055] S23. Build a training data set.

[0056] For each labeled street view image, regardless of the temporal relationship between street views, it is stored in three folders named after the three categories and named after its corresponding panoID. For street views with a temporal relationship, the street view temporal relationship is stored in a json file. When used later, if the categories corresponding to the two street views are different, a change is returned as 1, otherwise a change is returned as 0.

[0057] S24. Training the initial street view spatiotemporal feature extraction model.

[0058] The initial street view spatiotemporal feature extraction model is an improved FCOS model, which includes ConvNeXt-T Backbone, FPN network, spatiotemporal attention module and head module.

[0059] The FCOS network is used to capture spatial features in the image, such as beam and column components and surface materials of the building facade, and then the street view images are classified into four categories: initial reinforcement state, reinforcement process state, final reinforcement state, and no building;

[0060] The street view images in pairs of non-repeating groups in each street view sequence of the training set are input into the improved FCOS network under the dual input scenario. After the street view images are input, they will first pass through the ConvNeXt-T Backbone to generate multi-scale feature maps. FPN generates a more fine-grained multi-scale feature representation by integrating feature maps of different scales. Then, they are input into the spatiotemporal attention module to calculate the weight of the spatial position, and then the true value label of each feature point is assigned, including the target category, bounding box and centrality target. On this basis, the network performs classification, regression and centrality prediction, and calculates the loss function. Subsequently, the gradient backpropagation updates the network weights until the model converges and the training is stopped. At this time, the obtained model is determined to be the trained street view spatiotemporal feature extraction model.

[0061] S3. Obtain a street view sequence image to be identified, input the street view sequence image to be identified into a trained street view spatiotemporal feature extraction model, and determine the reinforcement state and change time point of the non-fortified building in the street view sequence image to be identified.

[0062] In a feasible implementation, all street view images in the street view sequence have a size of 4096×2048 pixels, including three RGB channels, and the input street view image is a street view image or a pair of street view images in the same street view sequence. It is denoted as C×H×W×N, where C represents the number of channels, H represents the pixel size of the image height, W represents the pixel size of the image width, and N represents the number of input images. In this application scenario, it is usually 1 or 2. This network has a total of 18 residual blocks with a stacking frequency of (3, 3, 9, 3), and all images are normalized for illumination to reduce the impact of illumination.

[0063] The street view spatiotemporal feature extraction model can be an improved FCOS network, and the improvements mainly include the following two points:

[0064] First, the ConvNeXt-T algorithm is used to replace the ResNet-50 algorithm used in the backbone part of the original FCOS network; the ConvNeXt-T algorithm introduces the Layer Norm layer in each convolution block to replace the batch normalization BatchNorm, uses the GELU activation function to improve the nonlinear expression ability of the network, and uses a larger 7x7 convolution kernel depth-separable convolution to replace the standard convolution in ResNet, thereby increasing the network's receptive field while reducing the number of parameters and calculations.

[0065] Second, a spatiotemporal attention module (STAM) is introduced before each head of the FCOS network; STAM (spatial-temporal attention module) combines information from the spatial and temporal dimensions to capture key changes in the input feature map. This module is introduced before each head of the improved FCOS network, which can effectively improve the ability to recognize building changes in street view sequence images.

[0066] Optionally, the street view spatiotemporal feature extraction model includes a ConvNeXt-T Backbone, an FPN network, a spatiotemporal attention module, and a head module;

[0067] The specific operation steps of S3 may include the following S31-S35:

[0068] S31, inputting the street scene sequence images to be identified into ConvNeXt-T Backbone, and obtaining a plurality of feature images after being processed by ConvNeXt-T Backbone.

[0069] Optionally, ConvNeXt-T Backbone includes: a first convolution layer Pre_conv, a Layer Norm module, a second convolution layer Conv_2, a ConvNeXt module, a third convolution layer Conv_3, a fourth convolution layer Conv_4 and a fifth convolution layer Conv_5; on this basis, the specific operation steps of S31 may include S311-S314:

[0070] S311, inputting the street scene sequence image to be recognized into the first convolution layer Pre_conv, and performing a normalization operation on the first image after the convolution through a Layer Norm module.

[0071] In a feasible implementation, the convolution kernel size is 4×4, the stride is 4, and after convolution, the street view size is 1024×512×96×N. Then, Layer Norm is used to normalize the channel dimension.

[0072] S312. Input the normalized first image into the second convolution layer Conv_2, connect the second convolution layer Conv_2 to the ConvNeXt module, and obtain the second image through the processing of the ConvNeXt module; the processing of the ConvNeXt module includes depthwise separable convolution, Layer Norm operation, 1x1 convolution, GELU activation function processing, 1x1 convolution, Layer Scale and DropPath to obtain a feature image, and add the input image and output image of the ConvNeXt module.

[0073] In a feasible implementation, Conv_2 (the second convolution layer) is connected to the ConvNeXt block structure, which consists of a convolution layer, a normalization layer, and an activation function. The number of channels of the input feature map is 96. First, a depth-separable convolution is performed with a convolution kernel size of 7x7 to keep the number of channels of the input and output consistent at 96. After the convolution operation, a Layer Norm operation is performed to standardize each channel. Next is a 1x1 convolution operation to increase the number of channels of the feature map from 96 to 384, and then the GELU activation function is applied to increase the nonlinear ability of the model. Another 1x1 convolution operation is performed to reduce the number of channels from 384 back to 96, and then Layer Scale is used to apply a learnable scale factor to the output after convolution for normalization, and then Drop Path is used to prevent overfitting and improve the generalization ability of the model. Finally, the initial input feature map is added to the processed feature map to realize the residual connection, and the number of residual blocks in this layer is 3.

[0074] S313, input the second image into the third convolution layer Conv_3, and obtain the first feature image after downsampling and ConvNeXt module processing.

[0075] In a feasible implementation, Conv_3 (the third convolution layer) first performs downsampling, applies LayerNorm to normalize each channel, and then sets the convolution kernel size to 2×2 and the stride to 2 for downsampling. After convolution, the street view size is 512×256×192×N. The ConvNeXt block structure is connected, which consists of a convolution layer, a normalization layer, and an activation function. The number of channels of the input feature map is 192. First, a depth-separable convolution is performed with a convolution kernel size of 7x7, keeping the number of input and output channels consistent at 192. After the convolution operation, a Layer Norm operation is performed to normalize each channel. Next is a 1x1 convolution operation to increase the number of channels of the feature map from 192 to 768, and then the GELU activation function is applied to increase the nonlinear ability of the model. A 1x1 convolution operation is performed again to reduce the number of channels from 384 to 96. Layer Scale is then used to apply a learnable scale factor to the convolution output for normalization. Drop Path is then used to prevent overfitting and improve the generalization ability of the model. Finally, the initial input feature map is added to the processed feature map to achieve a residual connection. The number of residual blocks in this layer is 3.

[0076] S314, input the first feature image to the fourth convolution layer Conv_4, and obtain the second feature image after downsampling and ConvNeXt module processing.

[0077] In a feasible implementation, Conv_4 (the fourth convolution layer) first performs downsampling, applies LayerNorm to normalize each channel, and then sets the convolution kernel size to 2×2 and the stride to 2 for downsampling. After convolution, the street view size is 256×128×384×N. The ConvNeXt block structure is connected, which consists of a convolution layer, a normalization layer, and an activation function. The number of channels of the input feature map is 384. First, a depth-separable convolution is performed with a convolution kernel size of 7x7, keeping the number of input and output channels consistent at 384. After the convolution operation, a Layer Norm operation is performed to normalize each channel. Next is a 1x1 convolution operation to increase the number of channels of the feature map from 384 to 1536, and then the GELU activation function is applied to increase the nonlinear ability of the model. A 1x1 convolution operation is performed again to reduce the number of channels from 1536 to 384. Layer Scale is then used to apply a learnable scale factor to the convolution output for normalization. Drop Path is then used to prevent overfitting and improve the generalization ability of the model. Finally, the initial input feature map is added to the processed feature map to achieve a residual connection. The number of residual blocks in this layer is 9.

[0078] S315, input the second feature image into the fifth convolution layer Conv_5, and obtain the third feature image after downsampling and ConvNeXt module processing.

[0079] In a feasible implementation, Conv_5 (the fifth convolution layer) first performs downsampling, applies LayerNorm to normalize each channel, and then sets the convolution kernel size to 2×2 and the stride to 2 for downsampling. After convolution, the street view size is 128×64×768×N. The ConvNeXt block structure is connected, which consists of a convolution layer, a normalization layer, and an activation function. The number of channels of the input feature map is 384. First, a depth-separable convolution is performed with a convolution kernel size of 7x7, keeping the number of input and output channels consistent at 768. After the convolution operation, a Layer Norm operation is performed to normalize each channel. Next is a 1x1 convolution operation to increase the number of channels of the feature map from 768 to 3072, and then the GELU activation function is applied to increase the nonlinear ability of the model. Perform another 1x1 convolution operation to reduce the number of channels from 3072 to 768, and then use Layer Scale to apply a learnable scale factor to the convolution output for normalization, and then use Drop Path to prevent overfitting and improve the generalization ability of the model. Finally, add the initial input feature map and the processed feature map to achieve residual connection, and the number of residual blocks in this layer is 3.

[0080] It should be noted that Conv_3 (the third convolution layer) corresponds to C3 of the original algorithm framework, Conv_4 (the fourth convolution layer) corresponds to C4 of the original algorithm framework, and Conv_5 (the fifth convolution layer) corresponds to C5 of the original algorithm framework.

[0081] S32. Input the multiple feature images output by ConvNeXt-T Backbone into the FPN network, and obtain output feature vectors after being processed by the FPN network.

[0082] In a feasible implementation, the Feature Pyramid step in FCOS is not improved compared to the original FCOS algorithm. The FPN network includes multiple Conv2d convolutional layers. The connections between the layers are introduced as follows:

[0083] like Figure 2As shown in the figure, the output feature of C5 is convolved by 1×1 to obtain P5; the output feature of C4 is convolved by 1×1 and then upsampled by 2 times with the output feature of C5 to obtain P4; similarly, the output feature of C3 is convolved by 1×1 and then upsampled by 2 times with the output feature of C4 to obtain P3. P6 and P7 are obtained by 1×1 convolution of P5 with a stride of 2. P3~P7 can be predicted after a 3×3 convolution, which is to reduce the aliasing effect caused by upsampling.

[0084] S33. Input the output feature vector into the spatiotemporal attention module, and obtain the feature matrix after being processed by the spatiotemporal attention module.

[0085] Optionally, the spatiotemporal attention module includes multiple 1×1 convolutional layers and a softmax activation function. On this basis, the specific operation steps of S33 may include S331-S333:

[0086] S331, performing two 1×1 convolutions on the output feature vectors, retaining the first channel number, and reshaping the convolved vector features into a first matrix and a second matrix respectively;

[0087] S332, performing 1×1 convolution on the output feature vector, retaining the second channel number and reshaping the convolved vector feature into a third matrix;

[0088] S333. Transpose the first matrix and multiply it with the second matrix, generate a similarity matrix according to the softmax activation function, multiply the third matrix and the similarity matrix to obtain a fourth matrix, add the fourth matrix and the output feature vector to obtain a feature matrix.

[0089] In a feasible implementation, the improved FCOS algorithm introduces a spatiotemporal attention module (STAM) before each head. The operation mechanism of the spatiotemporal attention module is introduced as follows:

[0090] Input feature vector from FPN part The dimension is , first of all Perform two 1×1 convolutions to obtain two sets of tensors (keys) and (queries), with dimensions , retain the number of channels Will and Reshape into a matrix and , the dimensions are ,in Including the height, width and number of street view images of the feature vector. Perform another 1×1 convolution to get the tensor (values), number of reserved channels Will Reshape into a matrix , the dimension is ,in Including the height, width and number of street view images of the feature vector. After transposition, the matrix Multiply, divide each element by And apply the softmax activation function to generate the similarity matrix , the dimension is . The matrix With the similarity matrix Multiply the output matrix , the dimension is , after reshaping, we get the matrix , the dimension is , and finally the matrix With the input matrix Add together to get the matrix , the dimension is At this point, the spatiotemporal attention mechanism module is completed.

[0091] S34, input the feature matrix into the head module to obtain the spatial features of the street scene sequence image to be identified.

[0092] In a feasible implementation manner, the head module in the embodiment of the present invention is consistent with the structure and operation mechanism of the original FCOS algorithm, and no improvement is made. The head module still includes Conv2d+GN+ReLU and multiple Conv2d convolutional layers. Therefore, the embodiment of the present invention will not be described in detail.

[0093] S35. Determine the reinforcement state and change time point of the non-fortified building in the street scene sequence image to be identified according to the spatial characteristics.

[0094] In a feasible implementation, the improved FCOS network is used to capture the spatial features in the image, such as the beam and column components of the building facade, the surface material, etc., and then the street view images are classified into four categories: the initial state of reinforcement, the state of reinforcement process, the final state of reinforcement, and no building. At the same time, the temporal features of the image pairs are captured, and then this group of street view images are classified into two categories: changed and unchanged.

[0095] S4. Combined with geographic information data, geo-code the identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain the thermal map of building reinforcement in the area of ​​interest over the years.

[0096] In a feasible implementation, the building footprint data of the concerned area is imported by using geographic information system technology, and the shooting point coordinates, vehicle head direction, true north direction and azimuth of the identified object in the street view are combined for positioning, and multiple street view images are combined for cross-validation to spatially locate and encode the reinforced buildings. The geocoded reinforced building data is analyzed, the number and distribution of reinforced buildings in different time periods are counted, the changing trend of the number and distribution of reinforced buildings is identified, and heat maps of reinforced building data in different periods are generated to show the density and distribution changes of building reinforcement in the concerned area.

[0097] In the embodiment of the present invention, a street view sequence data set containing temporal features is constructed, and the spatiotemporal features of the street view are extracted using an improved FCOS network model. The nonlinear expression ability of the network is improved by the ConvNeXt-T algorithm, and the receptive field of the network is increased, while the number of parameters and the amount of calculation are reduced. The information of the spatial and temporal dimensions is combined by the spatiotemporal attention module to capture the key changes in the input feature map. The module is introduced before each head of the improved FCOS network, which can effectively improve the recognition ability of building changes in street view sequence pictures. Furthermore, the reinforcement status of non-fortified buildings in different time periods can be effectively identified, and the change time points in the reinforcement process can be clearly identified. The efficiency of building safety management can be greatly improved and the investment of human resources can be reduced. Combining the results of street view image recognition with geographic information data, the reinforcement of non-fortified buildings can be accurately geocoded, and the spatial distribution of building reinforcement can be obtained, providing more accurate data support for subsequent urban planning and building safety assessment.

[0098] Figure 3 The block diagram of a device for identifying urban non-fortified building reinforcement based on historical street view pictures according to an exemplary embodiment is shown. The device is used in a method for identifying urban non-fortified building reinforcement based on historical street view pictures. Figure 3 The device includes a construction unit 310, a training unit 320, a recognition unit 330 and a determination unit 340. Among them:

[0099] A construction unit 310 is used to obtain historical street view images and street view metadata of a target area, and construct a street view sequence dataset containing temporal features according to the historical street view images and street view metadata;

[0100] A training unit 320 is used to annotate street view images of non-fortified buildings in different reinforcement states in the street view sequence data set to obtain a street view training set, and use the street view training set to train an initial street view spatiotemporal feature extraction model to obtain a trained street view spatiotemporal feature extraction model, wherein the street view spatiotemporal feature extraction model is an improved FCOS network;

[0101] The recognition unit 330 is used to obtain a street view sequence image to be recognized, input the street view sequence image to be recognized into a trained street view spatiotemporal feature extraction model, and determine the reinforcement state and change time point of the non-fortified building in the street view sequence image to be recognized;

[0102] The determination unit 340 is used to combine the geographic information data, geocode the identified buildings, obtain the spatial distribution and change trend of the reinforced buildings, and obtain the thermal map of the building reinforcement in the area of ​​interest over the years.

[0103] In the embodiment of the present invention, a street view sequence data set containing temporal features is constructed, and the spatiotemporal features of the street view are extracted using an improved FCOS network model. The nonlinear expression ability of the network is improved by the ConvNeXt-T algorithm, and the receptive field of the network is increased, while the number of parameters and the amount of calculation are reduced. The information of the spatial and temporal dimensions is combined by the spatiotemporal attention module to capture the key changes in the input feature map. The module is introduced before each head of the improved FCOS network, which can effectively improve the recognition ability of building changes in street view sequence pictures. Furthermore, the reinforcement status of non-fortified buildings in different time periods can be effectively identified, and the change time points in the reinforcement process can be clearly identified. The efficiency of building safety management can be greatly improved and the investment of human resources can be reduced. Combining the results of street view image recognition with geographic information data, the reinforcement of non-fortified buildings can be accurately geocoded, and the spatial distribution of building reinforcement can be obtained, providing more accurate data support for subsequent urban planning and building safety assessment.

[0104] Figure 4 is a schematic diagram of the structure of a device for identifying urban non-fortified building reinforcement based on historical street view images provided by an embodiment of the present invention. Figure 4 As shown, the urban non-fortified building reinforcement identification device based on historical street view pictures may include the above Figure 3 The device for identifying reinforcement of urban non-fortified buildings based on historical street view pictures is shown. Optionally, the device for identifying reinforcement of urban non-fortified buildings based on historical street view pictures 410 may include a first processor 2001 .

[0105] Optionally, the urban non-fortified building reinforcement identification device 410 based on historical street view pictures may also include a memory 2002 and a transceiver 2003 .

[0106] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0107] Combine the following Figure 4 The components of the urban non-fortified building reinforcement identification device 410 based on historical street view images are specifically introduced:

[0108] The first processor 2001 is the control center of the urban non-fortified building reinforcement identification device 410 based on historical street view images, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0109] Optionally, the first processor 2001 can perform various functions of the urban non-fortified building reinforcement identification device 410 based on historical street view images by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0110] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0111] In a specific implementation, as an embodiment, the urban non-fortified building reinforcement identification device 410 based on historical street view pictures may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0112] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0113] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the computer through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0114] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0115] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0116] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0117] It should be noted that Figure 4 The structure of the urban non-fortified building reinforcement identification device 410 based on historical street view images shown in the figure does not constitute a limitation on the router. The actual knowledge structure identification device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0118] In addition, the technical effects of the urban non-fortified building reinforcement identification device 410 based on historical street view images can refer to the technical effects of the urban non-fortified building reinforcement identification method based on historical street view images described in the above method embodiment, and will not be repeated here.

[0119] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0120] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0121] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0122] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0123] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0124] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0125] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0127] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0130] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0131] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for identifying urban non-fortified building reinforcement based on historical street view images, characterized in that: The method comprises: S1. Obtain historical street view images and street view metadata of a target area, and construct a street view sequence dataset containing temporal features according to the historical street view images and street view metadata; S2, annotating street view images of non-fortified buildings in different reinforcement states in the street view sequence data set to obtain a street view training set, and using the street view training set to train an initial street view spatiotemporal feature extraction model to obtain a trained street view spatiotemporal feature extraction model, wherein the street view spatiotemporal feature extraction model is an improved FCOS network; S3, obtaining a street view sequence image to be identified, inputting the street view sequence image to be identified into a trained street view spatiotemporal feature extraction model, and determining the reinforcement state and change time point of the non-fortified building in the street view sequence image to be identified; S4. Combined with geographic information data, geo-code the identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain the thermal map of building reinforcement in the area of ​​interest over the years.

2. The urban non-fortified building reinforcement identification method based on historical street view images according to claim 1 is characterized in that: The step S1 of acquiring historical street view images and street view metadata of the target area, and constructing a street view sequence dataset containing temporal features according to the historical street view images and street view metadata, includes: S11, obtaining a road network data file of the target area, and taking points at equal distances on the road network data file according to the selected specific sampling point density to determine the sampling point locations of the historical street view image; S12. For each sampling point location, obtain a latest updated street view picture with the shortest distance from the sampling point location; wherein the latest updated street view picture refers to a street view picture with the smallest absolute value of the difference between the shooting time and the current time; each street view picture corresponds to a unique panoID value, and the panoID value is used to indicate street view metadata, and the street view metadata includes the shooting point coordinates, shooting time, vehicle head direction, true north direction and historical street view column data corresponding to the street view picture; S13. According to the historical street view column data in the street view metadata indicated by the panoID value corresponding to each street view image, the panoID values ​​corresponding to the multiple street view images whose corresponding sampling point locations are in different historical periods are obtained; according to the panoID values ​​corresponding to the multiple street view images in different historical periods, the corresponding street view metadata are obtained; and according to the multiple street view images in different historical periods and the corresponding street view metadata, street view sequence data containing time series features are composed.

3. The urban non-fortified building reinforcement identification method based on historical street view pictures according to claim 1 is characterized in that: The street view sequence data containing temporal features is expressed by the following formula (1): （1） in, Represents a collection of street view sequence data containing time series features. Indicates the longitude of the shooting point coordinates in the street view metadata. Indicates the latitude of the shooting point coordinates in the street view metadata. represents the total number of street view images in the collection, Represents the i-th street view picture with the coordinates of the shooting point.

4. The urban non-fortified building reinforcement identification method based on historical street view images according to claim 1 is characterized in that: The different reinforcement states of S2 include three states: the reinforcement initial state, the reinforcement process state and the reinforcement final state; The initial state of reinforcement refers to the state where the non-fortified building has no obvious beam and column structure on the facade, the overall structure lacks maintenance, and the surface material has aged; The reinforcement process state refers to the state where the non-fortified building is in the state of construction equipment accompanied by scaffolding and coverings; The final state of reinforcement refers to the state in which the non-fortified building has an obvious beam-column structure on the facade, showing good maintenance and reinforcement effects.

5. The urban non-fortified building reinforcement identification method based on historical street view images according to claim 1 is characterized in that: The street view spatiotemporal feature extraction model includes a ConvNeXt-T Backbone, an FPN network, a spatiotemporal attention module, and a head module; The step S3 of inputting the street view sequence images to be identified into a trained street view spatiotemporal feature extraction model to determine the reinforcement state and change time point of the non-fortified buildings in the street view sequence images to be identified includes: S31, inputting the street scene sequence images to be identified into ConvNeXt-T Backbone, and obtaining a plurality of feature images after being processed by ConvNeXt-T Backbone; S32, inputting multiple feature images output by ConvNeXt-T Backbone into the FPN network, and obtaining output feature vectors after being processed by the FPN network; S33, input the output feature vector into the spatiotemporal attention module, and obtain a feature matrix after being processed by the spatiotemporal attention module; S34, inputting the feature matrix into the head module to obtain the spatial features of the street scene sequence image to be identified; S35. Determine the reinforcement state and change time point of the non-fortified building in the street view sequence image to be identified according to the spatial feature.

6. The urban non-fortified building reinforcement identification method based on historical street view images according to claim 5 is characterized in that: The ConvNeXt-T Backbone includes: a first convolution layer Pre_conv, a Layer Norm module, a second convolution layer Conv_2, a ConvNeXt module, a third convolution layer Conv_3, a fourth convolution layer Conv_4 and a fifth convolution layer Conv_5; The FPN network includes: a plurality of Conv2d convolutional layers; The spatiotemporal attention module includes: multiple 1×1 convolutional layers and a softmax activation function; The head module includes: Conv2d+GN+ReLU and multiple Conv2d convolutional layers.

7. The method for identifying urban non-fortified building reinforcement based on historical street view images according to claim 6, characterized in that: In S31, the street scene sequence image to be identified is input into ConvNeXt-T Backbone, and after being processed by ConvNeXt-T Backbone, a plurality of feature images are obtained, including: Input the street scene sequence image to be identified into the first convolution layer Pre_conv, and perform a normalization operation on the first image after convolution through a Layer Norm module; Input the normalized first image into the second convolution layer Conv_2, connect the second convolution layer Conv_2 to the ConvNeXt module, and obtain the second image through the processing of the ConvNeXt module; the processing of the ConvNeXt module includes depthwise separable convolution, Layer Norm operation, 1x1 convolution, GELU activation function processing, 1x1 convolution, Layer Scale and Drop Path to obtain a feature image, and add the input image and output image of the ConvNeXt module; The second image is input into the third convolution layer Conv_3, and processed by downsampling and ConvNeXt module to obtain the first feature image; The first feature image is input into the fourth convolution layer Conv_4, and processed by downsampling and ConvNeXt module to obtain the second feature image; The second feature image is input into the fifth convolution layer Conv_5, and after downsampling and ConvNeXt module processing, the third feature image is obtained.

8. The method for identifying urban non-fortified building reinforcement based on historical street view images according to claim 6, characterized in that: The S33 inputs the output feature vector into the spatiotemporal attention module, and after being processed by the spatiotemporal attention module, a feature matrix is ​​obtained, including: Perform two 1×1 convolutions on the output feature vectors, retain the first channel number, and reshape the convolved vector features into the first matrix and the second matrix respectively; Perform 1×1 convolution on the output feature vector, retain the second channel number and reshape the convolved vector feature into the third matrix; The first matrix is ​​transposed and multiplied with the second matrix, a similarity matrix is ​​generated according to the softmax activation function, the third matrix is ​​multiplied with the similarity matrix to obtain a fourth matrix, and the fourth matrix is ​​added to the output feature vector to obtain a feature matrix.

9. A device for identifying urban non-fortified building reinforcements based on historical street view images, the device for identifying urban non-fortified building reinforcements based on historical street view images is used to implement the urban non-fortified building reinforcement identification method based on historical street view images as claimed in any one of claims 1 to 8, characterized in that: The device comprises: A construction unit, configured to obtain historical street view images and street view metadata of a target area, and construct a street view sequence dataset containing temporal features according to the historical street view images and street view metadata; A training unit is used to annotate street view images of non-fortified buildings in different reinforcement states in the street view sequence data set to obtain a street view training set, and use the street view training set to train an initial street view spatiotemporal feature extraction model to obtain a trained street view spatiotemporal feature extraction model, wherein the street view spatiotemporal feature extraction model is an improved FCOS network; The recognition unit is used to obtain a street scene sequence image to be recognized, input the street scene sequence image to be recognized into a trained street scene spatiotemporal feature extraction model, and determine the reinforcement state and change time point of the non-fortified building in the street scene sequence image to be recognized; The determination unit is used to combine geographic information data, geocode the identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain the thermal map of building reinforcement in the area of ​​interest over the years.

10. A device for identifying urban non-fortified building reinforcement based on historical street view pictures, characterized in that: The urban non-fortified building reinforcement identification device based on historical street view pictures includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

Images