Urban non-fortified building reinforcement identification method and device based on historical street scene images

Abstract

The present invention discloses a method and device for identifying reinforced urban non-fortified buildings based on historical street scene images. The method relates to the fields of civil engineering and image processing. The method comprises: constructing a street scene sequence dataset containing temporal features based on historical street scene images and street scene metadata; annotating street scene images of non-fortified buildings in different reinforcement states to obtain a street scene training set; and training a street scene spatiotemporal feature extraction model, which is an improved FCOS network; inputting the street scene sequence images to be identified into the trained street scene spatiotemporal feature extraction model to determine the reinforcement state and change time point of the non-fortified buildings in the street scene sequence images; and geocoding the identified buildings in combination with geographic information data to obtain the spatial distribution and change trend of reinforced buildings, thereby obtaining a heat map of building reinforcement in the area of interest over the years. The present invention can significantly improve the efficiency of building safety management and reduce human resource investment.

Description

Technical Field

[0001] The present invention relates to the fields of civil engineering technology and image processing technology, and in particular to a method and device for identifying reinforced urban non-fortified buildings based on historical street scene images. Background Art

[0002] The reinforcement and maintenance of urban buildings are crucial for urban planning and disaster prevention and mitigation. However, traditional building reinforcement monitoring methods often rely on manual inspections and record-keeping, which is inefficient. As cities expand and the number of buildings increases, these traditional methods are becoming increasingly inadequate. Compared to traditional building reinforcement monitoring methods, a method for identifying reinforcement of non-fortified urban buildings based on historical streetscape images can efficiently monitor the reinforcement of non-fortified urban buildings on a large scale.

[0003] To identify reinforced non-fortified buildings across a large area, it's necessary to obtain historical street view images of the area of interest and construct a sample set of street view images. Computer technology can then be used to train a model capable of automatically identifying reinforced non-fortified buildings in urban areas. This allows for efficient and accurate large-scale identification of reinforced non-fortified buildings within street view images. Using geographic information technology, buildings in the street view can be geocoded based on metadata and building footprint data, accurately locating and recording the geographic location of reinforced buildings.

[0004] Prior Art 1: (Wang, C., Antos, SE, Triveno, LM, et al. Automatic detection of unreinforced masonry buildings from street view images using deep learning-based image segmentation[J]. Automation in Construction, 2021,132, 103968.) proposed an automatic detection method for unreinforced masonry buildings in street view images based on deep learning image segmentation. By manually annotating a street view image dataset and training a prediction model, unreinforced masonry buildings in street view images can be identified. It has also been proven that this method can overcome noise and occlusion problems. However, this method only focuses on building detection at a single time point and does not take into account changes in buildings over different time periods. Therefore, it is insufficient in identifying time series changes in building reinforcement.

[0005] Prior Art 2: (Jiang, S., Lin, H., Ren, H.; et al. MDANet: A High-Resolution City Change Detection Network Based on Difference and Attention Mechanisms under Multi-Scale Feature Fusion[J]. Remote Sens, 2024, 16, 1387.) proposed a high-resolution urban change detection network (MDANet) based on a multi-scale feature fusion and attention mechanism. This method effectively improves the accuracy of urban change detection by fusing multi-scale features and combining the difference and attention mechanisms. MDANet can detect subtle changes in high-resolution images and provides high spatiotemporal resolution. However, this method focuses primarily on urban change detection and does not address the specific identification of non-fortified building reinforcements, resulting in low efficiency and accuracy in this area. Summary of the Invention

[0006] To address the technical issues of low efficiency and accuracy in identifying specific non-fortified building reinforcements in existing technologies, the present invention provides a method and device for identifying urban non-fortified building reinforcements based on historical street scene images. The technical solution is as follows:

[0007] In one aspect, a method for identifying reinforced non-fortified buildings in cities based on historical street scene images is provided. The method is implemented by a device for identifying reinforced non-fortified buildings in cities based on historical street scene images. The method comprises:

[0008] S1. Obtain historical street view images and street view metadata of a target area, and construct a street view sequence dataset containing temporal features based on the historical street view images and street view metadata;

[0009] S2. Labeling street view images of non-fortified buildings in different reinforcement states in the street view sequence dataset to obtain a street view training set, and using the street view training set to train an initial street view spatiotemporal feature extraction model to obtain a trained street view spatiotemporal feature extraction model, wherein the street view spatiotemporal feature extraction model is an improved FCOS network;

[0010] S3. Obtain a sequence of street scene images to be identified, input the sequence of street scene images to be identified into a trained street scene spatiotemporal feature extraction model, and determine the reinforcement state and change time point of the non-fortified building in the sequence of street scene images to be identified;

[0011] S4. Combine geographic information data to geocode identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain the thermal map of building reinforcement in the area of interest over the years.

[0012] On the other hand, a device for identifying reinforcement of urban non-fortified buildings based on historical street scene images is provided. The device is applied to a method for identifying reinforcement of urban non-fortified buildings based on historical street scene images. The device comprises:

[0013] A construction unit, configured to obtain historical street view images and street view metadata of a target area, and construct a street view sequence dataset containing temporal features based on the historical street view images and street view metadata;

[0014] a training unit, configured to annotate street view images of non-fortified buildings in different reinforcement states in the street view sequence dataset to obtain a street view training set, and train an initial street view spatiotemporal feature extraction model using the street view training set to obtain a trained street view spatiotemporal feature extraction model, wherein the street view spatiotemporal feature extraction model is an improved FCOS network;

[0015] an identification unit, configured to obtain a sequence of street scene images to be identified, input the sequence of street scene images to be identified into a trained street scene spatiotemporal feature extraction model, and determine the reinforcement state and change time point of the non-fortified building in the sequence of street scene images to be identified;

[0016] The determination unit is used to combine geographic information data to geocode the identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain the thermal map of building reinforcement in the area of interest over the years.

[0017] On the other hand, a device for identifying urban non-fortified building reinforcements based on historical street view images is provided. The device for identifying urban non-fortified building reinforcements based on historical street view images comprises: a processor; a memory, wherein computer-readable instructions are stored on the memory. When the computer-readable instructions are executed by the processor, any one of the above-mentioned methods for identifying urban non-fortified building reinforcements based on historical street view images is implemented.

[0018] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for identifying reinforcement of urban non-fortified buildings based on historical street view images.

[0019] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0020] By constructing a street scene sequence dataset containing temporal features and extracting spatiotemporal features using an improved FCOS network model, the ConvNeXt-T algorithm improves the network's nonlinear representation and increases its receptive field, while reducing the number of parameters and computational overhead. A spatiotemporal attention module combines information from both spatial and temporal dimensions to capture key changes in the input feature map. This module, introduced before each head of the improved FCOS network, effectively enhances the ability to recognize building changes in street scene sequence images. Furthermore, it can effectively identify the reinforcement status of non-fortified buildings over different time periods and clearly identify the time points of changes during the reinforcement process. This significantly improves the efficiency of building safety management and reduces human resource investment. By combining the results of street scene image recognition with geographic information data, the reinforcement status of non-fortified buildings can be accurately geocoded, revealing the spatial distribution of building reinforcement, providing more accurate data support for subsequent urban planning and building safety assessments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of a method for identifying urban non-fortified building reinforcement based on historical street scene images provided by an embodiment of the present invention;

[0023] Figure 2 1 is a schematic diagram of the structure of an improved FCOS network provided by an embodiment of the present invention;

[0024] Figure 3 This is a block diagram of a device for identifying urban non-fortified building reinforcement based on historical street scene images provided by an embodiment of the present invention;

[0025] Figure 4 The present invention provides a schematic structural diagram of a device for identifying urban non-fortified building reinforcements based on historical street scene images. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0027] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0028] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0029] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0030] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0031] The embodiment of the present invention provides a method for identifying urban non-fortified building reinforcements based on historical street view images. The method can be implemented by an urban non-fortified building reinforcement identification device based on historical street view images. The urban non-fortified building reinforcement identification device based on historical street view images can be a terminal or a server. Figure 1 The flowchart of the method for identifying urban non-fortified building reinforcement based on historical street scene images is shown. The processing flow of the method may include the following steps:

[0032] S1. Obtain historical street view images and street view metadata of the target area, and construct a street view sequence dataset containing temporal features based on the historical street view images and street view metadata.

[0033] Optionally, the step of S1 includes S11-S13:

[0034] S11. Obtain a road network data file of the target area, and select points at equal distances on the road network data file according to the selected specific sampling point density to determine the sampling point locations of the historical street view image.

[0035] S12. For each sampling point location, obtain a newly updated street view image that is the closest to the sampling point location; the newly updated street view image is the street view image with the smallest absolute difference between the capture time and the current time; each street view image corresponds to a unique panoID value, and the panoID value is used to indicate street view metadata, which includes the capture point coordinates, capture time, vehicle head direction, true north direction, and historical street view column data corresponding to the street view image.

[0036] S13. Based on the historical street view column data in the street view metadata indicated by the panoID value corresponding to each street view image, obtain the panoID values corresponding to multiple street view images with corresponding sampling points in different historical periods; obtain the corresponding street view metadata based on the panoID values corresponding to the multiple street view images in different historical periods; and assemble street view sequence data containing time series features based on the multiple street view images in different historical periods and the corresponding street view metadata.

[0037] In one feasible implementation, first, a road network data file (shapefile format) of the area of interest is obtained from OpenStreetMap. A specific sampling point density, such as 10m, 25m, 50m, etc., is selected to determine the sampling point locations of the street view image. The attributes of each sampling point include latitude and longitude coordinates.

[0038] Secondly, for each determined sampling point location, the nearest and most recently updated street view panoramic image can be obtained (i.e., using the Baidu Maps official API, the latest street view image closest to the point can be obtained based on the latitude and longitude coordinates). Each street view image corresponds to a unique panoID value, and based on this unique panoID value, street view metadata such as the coordinates of the street view shooting point, the time of the street view shooting, the vehicle's heading, the true north direction, and the historical street view column can be obtained;

[0039] Then, for the historical street view column in the street view metadata, the panoID values corresponding to multiple street view images of different historical periods belonging to the sampling point can be obtained, and then the respective street view metadata can be obtained to form a street view sequence containing temporal features.

[0040] Optionally, the street view sequence data containing temporal features is expressed using the following formula (1):

[0041] (1)

[0042] in, Represents a collection of street view sequence data containing time series features. Indicates the longitude of the shooting point coordinates in the street view metadata. Indicates the latitude of the shooting point coordinates in the street view metadata. represents the total number of street view images in the collection, Represents the i-th street view picture with the coordinates of the shooting point.

[0043] The larger the index in the upper right corner, the closer the shooting time of the street view sequence image is to the present;

[0044] Therefore, each street view picture can be represented in a symbolic way.

[0045] S2. Label street view images of non-fortified buildings in different reinforcement states in the street view sequence dataset to obtain a street view training set. Use the street view training set to train the initial street view spatiotemporal feature extraction model to obtain a trained street view spatiotemporal feature extraction model. The street view spatiotemporal feature extraction model is an improved FCOS network.

[0046] Optionally, the process of training the initial street view spatiotemporal feature extraction model in S2 may include the following steps S21-S24:

[0047] S21. Label street view images of non-fortified buildings in different reinforcement states in the street view sequence dataset to obtain street view training data.

[0048] Among them, different reinforcement states include the initial state of reinforcement, the state of reinforcement process and the final state of reinforcement;

[0049] Among them, the initial state of reinforcement refers to the state where the non-fortified building has no obvious beam and column structure on the facade, the overall structure lacks maintenance, and the surface materials have aged;

[0050] The reinforcement process state refers to the state where the non-fortified building is in the state of construction equipment accompanied by scaffolding and covering;

[0051] The final state of reinforcement refers to the state in which the non-fortified building has an obvious beam-column structure on the facade, showing good maintenance and reinforcement effects.

[0052] In a feasible implementation, professionals label street view images containing non-fortified buildings in three states and classify them into an initial reinforcement state, a reinforcement process state, and a final reinforcement state to form street view training data.

[0053] S22. Preprocess the street view training data.

[0054] In a feasible implementation, all street view images in the street view sequence have a size of 4096×2048 pixels, including three RGB channels, and all images are subjected to illumination normalization processing to reduce illumination effects.

[0055] S23. Build a training dataset.

[0056] Each annotated street view image is stored in three folders named after the three categories, regardless of their temporal relationships. For street views with temporal relationships, this relationship is stored in a JSON file. When used later, if two street views correspond to different categories, a 1 is returned to indicate a change; otherwise, a 0 is returned to indicate no change.

[0057] S24. Train the initial street view spatiotemporal feature extraction model.

[0058] The initial street view spatiotemporal feature extraction model is the improved FCOS model, which includes a ConvNeXt-T backbone, an FPN network, a spatiotemporal attention module, and a head module.

[0059] The FCOS network is used to capture spatial features in images, such as building facade beams and columns, and surface materials, and then classify street view images into four categories: initial reinforcement state, reinforcement process state, final reinforcement state, and no building.

[0060] Pairs of unique street view images from each street view sequence in the training set are fed into the improved FCOS network for the dual-input scenario. After the street view images are fed into the ConvNeXt-T backbone, they generate multi-scale feature maps. The FPN integrates the feature maps at different scales to generate a finer-grained multi-scale feature representation. These images are then fed into the spatiotemporal attention module to calculate spatial position weights. The network then assigns a true value label to each feature point, including the object category, bounding box, and center target. Based on this, the network performs classification, regression, and center prediction, and calculates the loss function. Gradient backpropagation then updates the network weights until the model converges, at which point training is terminated. The resulting model is considered the trained street view spatiotemporal feature extraction model.

[0061] S3. Obtain a sequence of street view images to be identified, input the sequence of street view images to be identified into a trained street view spatiotemporal feature extraction model, and determine the reinforcement state and change time point of the non-fortified buildings in the sequence of street view images to be identified.

[0062] In one feasible implementation, all street view images in the street view sequence are sized 4096×2048 pixels, including RGB channels. The input street view image is either a single street view image or a pair of street view images from the same street view sequence. This is denoted as C×H×W×N, where C represents the number of channels, H represents the image height in pixels, W represents the image width in pixels, and N represents the number of input images, which is typically 1 or 2 in this application scenario. The network has 18 residual blocks stacked in the order of (3, 3, 9, 3). All images are subjected to illumination normalization to reduce the effects of lighting.

[0063] The street view spatiotemporal feature extraction model can be an improved FCOS network. The improvements mainly include the following two points:

[0064] First, the ConvNeXt-T algorithm was used to replace the ResNet-50 algorithm used in the backbone of the FCOS network. The ConvNeXt-T algorithm introduced a Layer Norm layer in each convolution block to replace BatchNorm, used the GELU activation function to improve the nonlinear expression capability of the network, and used a larger 7x7 depthwise separable convolution kernel to replace the standard convolution in ResNet, thereby increasing the network's receptive field while reducing the number of parameters and computational complexity.

[0065] Second, a spatiotemporal attention module (STAM) is introduced before each head of the FCOS network. The STAM combines information from both spatial and temporal dimensions to capture key changes in the input feature map. This module, introduced before each head of the improved FCOS network, effectively improves the ability to recognize building changes in street view image sequences.

[0066] Optionally, the street view spatiotemporal feature extraction model includes a ConvNeXt-T Backbone, an FPN network, a spatiotemporal attention module, and a head module;

[0067] The specific operation steps of S3 may include the following S31-S35:

[0068] S31. Input the street scene sequence image to be recognized into ConvNeXt-T Backbone. After being processed by ConvNeXt-T Backbone, a plurality of feature images are obtained.

[0069] Optionally, the ConvNeXt-T Backbone includes: a first convolution layer Pre_conv, a Layer Norm module, a second convolution layer Conv_2, a ConvNeXt module, a third convolution layer Conv_3, a fourth convolution layer Conv_4, and a fifth convolution layer Conv_5; on this basis, the specific operation steps of S31 may include S311-S314:

[0070] S311: Input the street scene sequence image to be recognized into the first convolution layer Pre_conv, and perform a normalization operation on the first image after convolution through the Layer Norm module.

[0071] In one feasible implementation, the convolution kernel size is 4×4 and the stride is 4. After convolution, the street view size is 1024×512×96×N. The channel dimension is then normalized by Layer Norm.

[0072] S312. Input the normalized first image into the second convolution layer Conv_2, connect the second convolution layer Conv_2 to the ConvNeXt module, and obtain the second image through the processing of the ConvNeXt module; the processing of the ConvNeXt module includes depth-separable convolution, Layer Norm operation, 1x1 convolution, GELU activation function processing, 1x1 convolution, Layer Scale and DropPath to obtain a feature image, and add the input image and output image of the ConvNeXt module.

[0073] In a feasible implementation, Conv_2 (the second convolution layer) is connected to the ConvNeXt block structure, which consists of a convolution layer, a normalization layer, and an activation function. The number of channels of the input feature map is 96. First, a depth-wise separable convolution is performed with a convolution kernel size of 7x7 to keep the number of channels of the input and output consistent at 96. After the convolution operation, a Layer Norm operation is performed to normalize each channel. Next, a 1x1 convolution operation is performed to increase the number of channels of the feature map from 96 to 384, and then the GELU activation function is applied to increase the nonlinear ability of the model. Another 1x1 convolution operation is performed to reduce the number of channels from 384 back to 96, and then Layer Scale is used to apply a learnable scale factor to the convolution output for normalization, and then Drop Path is used to prevent overfitting and improve the generalization ability of the model. Finally, the initial input feature map is added to the processed feature map to realize the residual connection. The number of residual blocks in this layer is 3.

[0074] S313: Input the second image into the third convolution layer Conv_3, and obtain the first feature image after downsampling and ConvNeXt module processing.

[0075] In a feasible implementation, Conv_3 (the third convolution layer) first performs downsampling, applies LayerNorm to normalize each channel, and then sets the convolution kernel size to 2×2 and the stride to 2 for downsampling. After convolution, the street view size is 512×256×192×N. The ConvNeXt block structure is connected, which consists of a convolution layer, a normalization layer, and an activation function. The number of channels of the input feature map is 192. First, a depth-wise separable convolution is performed with a convolution kernel size of 7x7, keeping the number of input and output channels consistent at 192. After the convolution operation, a Layer Norm operation is performed to normalize each channel. Next is a 1x1 convolution operation to increase the number of channels of the feature map from 192 to 768, and then the GELU activation function is applied to increase the nonlinear ability of the model. Another 1x1 convolution is performed to reduce the number of channels from 384 to 96. Layer Scale is then used to normalize the convolutional output by applying a learnable scaling factor. Drop Path is then used to prevent overfitting and improve the model's generalization. Finally, the initial input feature map is added to the processed feature map to implement a residual connection. This layer has three residual blocks.

[0076] S314: Input the first feature image into the fourth convolution layer Conv_4, and obtain the second feature image through downsampling and ConvNeXt module processing.

[0077] In a feasible implementation, Conv_4 (the fourth convolution layer) first performs downsampling, applies LayerNorm to normalize each channel, and then sets the convolution kernel size to 2×2 and the stride to 2 for downsampling. After convolution, the street view size is 256×128×384×N. The ConvNeXt block structure is connected, which consists of a convolution layer, a normalization layer, and an activation function. The number of channels of the input feature map is 384. First, a depth-wise separable convolution is performed with a convolution kernel size of 7x7, keeping the number of input and output channels consistent at 384. After the convolution operation, a LayerNorm operation is performed to normalize each channel. Next is a 1x1 convolution operation to increase the number of channels of the feature map from 384 to 1536, and then the GELU activation function is applied to increase the nonlinear ability of the model. Another 1x1 convolution is performed to reduce the number of channels from 1536 to 384. Layer Scale is then used to normalize the convolutional output by applying a learnable scaling factor. Drop Path is then used to prevent overfitting and improve the model's generalization. Finally, the initial input feature map is added to the processed feature map to implement a residual connection. This layer has nine residual blocks.

[0078] S315. Input the second feature image into the fifth convolution layer Conv_5, and obtain the third feature image after downsampling and ConvNeXt module processing.

[0079] In a feasible implementation, Conv_5 (the fifth convolution layer) first performs downsampling, applies LayerNorm to normalize each channel, and then sets the convolution kernel size to 2×2 and the stride to 2 for downsampling. After convolution, the street view size is 128×64×768×N. The ConvNeXt block structure is connected, which consists of a convolution layer, a normalization layer, and an activation function. The number of channels of the input feature map is 384. First, a depth-wise separable convolution is performed with a convolution kernel size of 7x7, keeping the number of input and output channels consistent at 768. After the convolution operation, a LayerNorm operation is performed to normalize each channel. Next is a 1x1 convolution operation to increase the number of channels of the feature map from 768 to 3072, and then the GELU activation function is applied to increase the nonlinear ability of the model. Another 1x1 convolution is performed to reduce the number of channels from 3072 to 768. Layer Scale is then used to normalize the convolutional output by applying a learnable scaling factor. Drop Path is then used to prevent overfitting and improve the model's generalization. Finally, the initial input feature map is added to the processed feature map to implement a residual connection. This layer has three residual blocks.

[0080] It should be noted that Conv_3 (the third convolution layer) corresponds to C3 of the original algorithm framework, Conv_4 (the fourth convolution layer) corresponds to C4 of the original algorithm framework, and Conv_5 (the fifth convolution layer) corresponds to C5 of the original algorithm framework.

[0081] S32. Input the multiple feature images output by ConvNeXt-T Backbone into the FPN network, and obtain output feature vectors after being processed by the FPN network.

[0082] In a feasible implementation, the Feature Pyramid step in FCOS is not improved compared to the original FCOS algorithm. The FPN network includes multiple Conv2d convolutional layers. The connections between the layers are introduced as follows:

[0083] like Figure 2As shown in the figure, the output features of C5 undergo a 1×1 convolution to obtain P5; the output features of C4 undergo a 1×1 convolution, then are upsampled by a factor of 2 and added to the output features of C5 to obtain P4; similarly, the output features of C3 undergo a 1×1 convolution, then are upsampled by a factor of 2 and added to the output features of C4 to obtain P3. P6 and P7 are derived from P5 through a 1×1 convolution with a stride of 2. P3-P7 all undergo a 3×3 convolution before prediction. The 3×3 convolution is used to reduce the aliasing effect caused by upsampling.

[0084] S33. Input the output feature vector into the spatiotemporal attention module, and obtain the feature matrix after being processed by the spatiotemporal attention module.

[0085] Optionally, the spatiotemporal attention module includes multiple 1×1 convolutional layers and a softmax activation function. On this basis, the specific operation steps of S33 may include S331-S333:

[0086] S331, perform two 1×1 convolutions on the output feature vector, retain the first channel number, and reshape the convolved vector features into a first matrix and a second matrix respectively;

[0087] S332, perform 1×1 convolution on the output feature vector, retain the second channel number, and reshape the convolved vector feature into a third matrix;

[0088] S333. Transpose the first matrix and multiply it by the second matrix, generate a similarity matrix according to the softmax activation function, multiply the third matrix by the similarity matrix to obtain a fourth matrix, add the fourth matrix to the output eigenvector to obtain a feature matrix.

[0089] In one feasible implementation, the improved FCOS algorithm introduces a spatiotemporal attention module (STAM) before each head. The following describes the operation mechanism of the spatiotemporal attention module:

[0090] Input feature vector from FPN part The dimension is , first of all Perform two 1×1 convolutions to obtain two sets of tensors (keys) and (queries), the dimensions are , retain the number of channels Will and Reshape into a matrix and , the dimensions are ,in Including the height, width and number of street view images of the feature vector. Perform another 1×1 convolution to get the tensor (values), number of reserved channels Will Reshape into a matrix , the dimension is ,in Including the height, width and number of street view images of the feature vector. After transposition, the matrix Multiply, divide each element by And apply the softmax activation function to generate the similarity matrix , the dimension is . The matrix and similarity matrix Multiply output matrix , the dimension is , after reshaping, we get the matrix , the dimension is , and finally the matrix With the input matrix Add up to get the matrix , the dimension is At this point, the spatiotemporal attention mechanism module is completed.

[0091] S34. Input the feature matrix into the head module to obtain the spatial features of the street scene sequence image to be identified.

[0092] In one feasible implementation, the head module in the embodiment of the present invention is consistent with the structure and operating mechanism of the original FCOS algorithm, without any improvement. The head module still includes Conv2d+GN+ReLU and multiple Conv2d convolutional layers. Therefore, this embodiment of the present invention will not be described in detail.

[0093] S35. Determine the reinforcement state and change time point of the non-fortified building in the street scene sequence image to be identified based on the spatial characteristics.

[0094] In one feasible implementation, an improved FCOS network is used to capture spatial features in images, such as building facade beams and columns, and surface materials, to classify street view images into four categories: initial reinforcement state, reinforcement process state, final reinforcement state, and no building. Temporal features of image pairs are also captured to classify these street view images into two categories: those with changes or those without changes.

[0095] S4. Combine geographic information data to geocode identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain the thermal map of building reinforcement in the area of interest over the years.

[0096] One feasible implementation utilizes geographic information system technology to import building footprint data for the area of interest. This data is then combined with the shooting point coordinates, vehicle heading, and true north direction from street view image metadata, along with the azimuth of the identified object in the street view. This data is then cross-validated across multiple street view images to spatially locate and encode reinforced buildings. This geocoded reinforced building data is analyzed to determine the number and distribution of reinforced buildings over different time periods, identifying trends in their number and distribution. Heat maps of the reinforced building data over different time periods are then generated to demonstrate the density and distribution of building reinforcement within the area of interest.

[0097] In this embodiment of the present invention, a street scene sequence dataset containing temporal features is constructed and an improved FCOS network model is used to extract the spatiotemporal features of the street scenes. The ConvNeXt-T algorithm is used to enhance the network's nonlinear representation capability and increase its receptive field, while reducing the number of parameters and computational complexity. A spatiotemporal attention module combines information from both spatial and temporal dimensions to capture key changes in the input feature map. This module, introduced before each head of the improved FCOS network, effectively enhances the ability to recognize building changes in street scene sequence images. Furthermore, it can effectively identify the reinforcement status of non-fortified buildings over different time periods and clearly identify the time points of changes during the reinforcement process. This significantly improves the efficiency of building safety management and reduces human resource investment. By combining the results of street scene image recognition with geographic information data, the reinforcement status of non-fortified buildings can be accurately geocoded, revealing the spatial distribution of building reinforcement, providing more accurate data support for subsequent urban planning and building safety assessments.

[0098] Figure 3 This is a block diagram of a device for identifying urban non-fortified building reinforcement based on historical street view images according to an exemplary embodiment. The device is used in a method for identifying urban non-fortified building reinforcement based on historical street view images. Figure 3 The device includes a construction unit 310, a training unit 320, a recognition unit 330, and a determination unit 340.

[0099] A construction unit 310 is configured to obtain historical street view images and street view metadata of a target area, and construct a street view sequence dataset containing temporal features based on the historical street view images and street view metadata;

[0100] a training unit 320 configured to annotate street view images of non-fortified buildings in different reinforcement states in the street view sequence dataset to obtain a street view training set, and to train an initial street view spatiotemporal feature extraction model using the street view training set to obtain a trained street view spatiotemporal feature extraction model, wherein the street view spatiotemporal feature extraction model is an improved FCOS network;

[0101] The recognition unit 330 is configured to obtain a sequence of street scene images to be recognized, input the sequence of street scene images to be recognized into a trained street scene spatiotemporal feature extraction model, and determine the reinforcement state and change time point of the non-fortified buildings in the sequence of street scene images to be recognized;

[0102] The determination unit 340 is used to combine geographic information data to perform geocoding on the identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain a thermal map of building reinforcement in the area of interest over the years.

[0103] In this embodiment of the present invention, a street scene sequence dataset containing temporal features is constructed and an improved FCOS network model is used to extract the spatiotemporal features of the street scenes. The ConvNeXt-T algorithm is used to enhance the network's nonlinear representation capability and increase its receptive field, while reducing the number of parameters and computational complexity. A spatiotemporal attention module combines information from both spatial and temporal dimensions to capture key changes in the input feature map. This module, introduced before each head of the improved FCOS network, effectively enhances the ability to recognize building changes in street scene sequence images. Furthermore, it can effectively identify the reinforcement status of non-fortified buildings over different time periods and clearly identify the time points of changes during the reinforcement process. This significantly improves the efficiency of building safety management and reduces human resource investment. By combining the results of street scene image recognition with geographic information data, the reinforcement status of non-fortified buildings can be accurately geocoded, revealing the spatial distribution of building reinforcement, providing more accurate data support for subsequent urban planning and building safety assessments.

[0104] Figure 4 Schematic diagram of a device for identifying urban non-fortified building reinforcement based on historical street scene images provided by an embodiment of the present invention. Figure 4 As shown, the urban non-fortified building reinforcement identification device based on historical street view pictures may include the above Figure 3 The apparatus for identifying reinforced non-fortified buildings in cities based on historical street view images is shown. Optionally, the apparatus for identifying reinforced non-fortified buildings in cities based on historical street view images 410 may include a first processor 2001 .

[0105] Optionally, the urban non-fortified building reinforcement identification device 410 based on historical street view pictures may further include a memory 2002 and a transceiver 2003 .

[0106] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0107] The following combination Figure 4 The components of the urban non-fortified building reinforcement identification device 410 based on historical street scene images are described in detail:

[0108] The first processor 2001 is the control center of the urban non-fortified building reinforcement identification device 410 based on historical street scene images. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs) or one or more field programmable gate arrays (FPGAs).

[0109] Optionally, the first processor 2001 can perform various functions of the urban non-fortified building reinforcement identification device 410 based on historical street view pictures by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0110] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0111] In a specific implementation, as an embodiment, the urban non-fortified building reinforcement identification device 410 based on historical street view images may also include multiple processors, such as Figure 4 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0112] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0113] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and access the memory 2002 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0114] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0115] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0116] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and may be connected to the device 410 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0117] It should be noted that Figure 4 The structure of the urban non-fortified building reinforcement identification device 410 based on historical street view images shown in the figure does not constitute a limitation on the router. The actual knowledge structure identification device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0118] In addition, the technical effects of the urban non-fortified building reinforcement identification device 410 based on historical street view images can refer to the technical effects of the urban non-fortified building reinforcement identification method based on historical street view images described in the above method embodiment, and will not be repeated here.

[0119] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0120] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0121] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0122] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0123] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0124] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0125] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0126] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0127] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0129] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0130] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0131] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for identifying urban non-fortified building reinforcement based on historical street scene images, characterized in that: The method comprises: S1. Obtain historical street view images and street view metadata of a target area, and construct a street view sequence dataset containing temporal features based on the historical street view images and street view metadata; S2. Labeling street view images of non-fortified buildings in different reinforcement states in the street view sequence dataset to obtain a street view training set, and using the street view training set to train an initial street view spatiotemporal feature extraction model to obtain a trained street view spatiotemporal feature extraction model, wherein the street view spatiotemporal feature extraction model is an improved FCOS network; S3. Obtain a sequence of street scene images to be identified, input the sequence of street scene images to be identified into a trained street scene spatiotemporal feature extraction model, and determine the reinforcement state and change time point of the non-fortified building in the sequence of street scene images to be identified; S4. Combine geographic information data to geocode identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain a thermal map of building reinforcement in the area of interest over the years; The street view spatiotemporal feature extraction model includes a ConvNeXt-T Backbone, an FPN network, a spatiotemporal attention module, and a head module. The S3 includes: S31, inputting the street scene sequence image to be identified into ConvNeXt-T Backbone, and obtaining a plurality of feature images after being processed by ConvNeXt-T Backbone; S32, inputting multiple feature images output by ConvNeXt-T Backbone into the FPN network, and obtaining output feature vectors after being processed by the FPN network; S33, input the output feature vector into the spatiotemporal attention module, and obtain a feature matrix after being processed by the spatiotemporal attention module; S34, inputting the feature matrix into the head module to obtain the spatial features of the street scene sequence image to be identified; S35. Determine the reinforcement state and the change time point of the non-fortified building in the street scene sequence image to be identified according to the spatial feature.

2. The urban non-fortified building reinforcement identification method based on historical street scene images according to claim 1 is characterized in that: The step S1 of obtaining historical street view images and street view metadata of a target area, and constructing a street view sequence dataset containing temporal features based on the historical street view images and street view metadata, includes: S11, obtaining a road network data file of the target area, and taking points at equal distances on the road network data file according to the selected specific sampling point density to determine the sampling point locations of the historical street view image; S12. For each sampling point location, obtain a newly updated street view image that is the closest to the sampling point location; wherein the newly updated street view image is the street view image with the smallest absolute difference between the capture time and the current time; each street view image corresponds to a unique panoID value, and the panoID value is used to indicate street view metadata, which includes the capture point coordinates, capture time, vehicle head direction, true north direction, and historical street view column data corresponding to the street view image; S13. Based on the historical street view column data in the street view metadata indicated by the panoID value corresponding to each street view image, obtain the panoID values corresponding to multiple street view images with corresponding sampling points in different historical periods; obtain the corresponding street view metadata based on the panoID values corresponding to the multiple street view images in different historical periods; and assemble street view sequence data containing time series features based on the multiple street view images in different historical periods and the corresponding street view metadata.

3. The urban non-fortified building reinforcement identification method based on historical street scene images according to claim 1 is characterized in that: The street view sequence data containing temporal features is expressed by the following formula (1): Among them, SV (lon，lat) Represents a collection of street view sequence data containing time series features, lon represents the longitude of the shooting point coordinates in the street view metadata, lat represents the latitude of the shooting point coordinates in the street view metadata, and n represents the total number of street view images in the collection. Represents the i-th street view picture with the coordinates of the shooting point.

4. The method for identifying urban non-fortified building reinforcement based on historical street scene images according to claim 1, characterized in that: The different reinforcement states of S2 include the initial reinforcement state, the reinforcement process state and the final reinforcement state; The initial state of reinforcement refers to the state where the facade of a non-fortified building has no obvious beam and column structure, the overall structure lacks maintenance, and the surface materials have aged; The reinforcement process state refers to the state where the non-fortified building is in the state of construction equipment accompanied by scaffolding and coverings; The final reinforcement state refers to the state in which the non-fortified building has an obvious beam-column structure on the facade, showing good maintenance and reinforcement effects.

5. The urban non-fortified building reinforcement identification method based on historical street scene images according to claim 1 is characterized in that: The ConvNeXt-T Backbone includes: a first convolution layer Pre_conv, a Layer Norm module, a second convolution layer Conv_2, a ConvNeXt module, a third convolution layer Conv_3, a fourth convolution layer Conv_4, and a fifth convolution layer Conv_5; The FPN network includes: multiple Conv2d convolutional layers; The spatiotemporal attention module includes: multiple 1×1 convolutional layers and a softmax activation function; The head module includes: Conv2d+GN+ReLU and multiple Conv2d convolutional layers.

6. The method for identifying urban non-fortified building reinforcement based on historical street scene images according to claim 5 is characterized in that: In step S31, the street scene sequence image to be identified is input into the ConvNeXt-T Backbone. After being processed by the ConvNeXt-T Backbone, a plurality of feature images are obtained, including: Input the street scene sequence image to be recognized into the first convolution layer Pre_conv, and perform a normalization operation on the convolved first image through the Layer Norm module; Input the normalized first image into the second convolution layer Conv_2, connect the second convolution layer Conv_2 to the ConvNeXt module, and obtain the second image through processing by the ConvNeXt module; the processing of the ConvNeXt module includes depthwise separable convolution, Layer Norm operation, 1x1 convolution, GELU activation function processing, 1x1 convolution, Layer Scale and Drop Path to obtain a feature image, and add the input image and output image of the ConvNeXt module; The second image is input into the third convolution layer Conv_3, and processed by downsampling and ConvNeXt module to obtain the first feature image; The first feature image is input into the fourth convolution layer Conv_4, and processed by downsampling and ConvNeXt module to obtain the second feature image; The second feature image is input into the fifth convolution layer Conv_5, and processed by downsampling and ConvNeXt module to obtain the third feature image.

7. The method for identifying urban non-fortified building reinforcement based on historical street scene images according to claim 5, characterized in that: The output feature vector of S33 is input into the spatiotemporal attention module. After being processed by the spatiotemporal attention module, a feature matrix is obtained, including: Perform two 1×1 convolutions on the output feature vectors, retain the first channel number, and reshape the convolved vector features into the first matrix and the second matrix respectively; Perform 1×1 convolution on the output feature vector, retain the second channel number, and reshape the convolved vector feature into the third matrix; The first matrix is transposed and multiplied with the second matrix to generate a similarity matrix according to the softmax activation function. The third matrix is multiplied with the similarity matrix to obtain a fourth matrix. The fourth matrix is added to the output eigenvector to obtain a feature matrix.

8. A device for identifying reinforced non-fortified buildings in cities based on historical street view images, wherein the device is used to implement the method for identifying reinforced non-fortified buildings in cities based on historical street view images as claimed in any one of claims 1 to 7, characterized in that: The device comprises: A construction unit, configured to obtain historical street view images and street view metadata of a target area, and construct a street view sequence dataset containing temporal features based on the historical street view images and street view metadata; a training unit, configured to annotate street view images of non-fortified buildings in different reinforcement states in the street view sequence dataset to obtain a street view training set, and train an initial street view spatiotemporal feature extraction model using the street view training set to obtain a trained street view spatiotemporal feature extraction model, wherein the street view spatiotemporal feature extraction model is an improved FCOS network; an identification unit, configured to obtain a sequence of street scene images to be identified, input the sequence of street scene images to be identified into a trained street scene spatiotemporal feature extraction model, and determine the reinforcement state and change time point of the non-fortified building in the sequence of street scene images to be identified; The determination unit is used to combine geographic information data to geocode the identified buildings, obtain the spatial distribution and change trend of reinforced buildings, and obtain the thermal map of building reinforcement in the area of interest over the years.

9. A device for identifying urban non-fortified building reinforcement based on historical street scene images, characterized in that: The urban non-fortified building reinforcement identification device based on historical street view images includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images