A method for evaluating urban skylines based on portable devices and deep learning
By using portable devices and deep learning methods to collect and analyze urban street view images, a comprehensive evaluation index for skylines is constructed, which solves the problem of unrealistic skyline contour extraction in existing technologies and achieves accurate quantitative assessment and intelligent guidance of urban skylines.
Patent Information
- Application Number
- CN202410024070.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-08
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-01-08
AI Technical Summary
Existing technologies struggle to quantitatively evaluate urban skylines from a human perspective, resulting in unrealistic skyline contour extraction and a lack of effective quantitative descriptive indicators.
The study uses portable devices to collect urban street view images, employs deep learning technology to identify street view elements and extract skyline contours, and constructs a comprehensive evaluation index system, including indicators such as tortuosity, total length, height difference, and distance difference, for quantitative evaluation.
It has enabled the evaluation of the authenticity and accuracy of urban skylines, reduced data collection and learning costs, improved the intelligence and accuracy of the evaluation, and provided scientific guidance for urban renewal and landscape transformation.
Smart Images

Figure CN117726615B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of urban landscape evaluation, and more particularly to an urban skyline evaluation method based on portable devices and deep learning. Background Technology
[0002] In urban design, skyline assessment is a crucial aspect of design and control. Skyline assessment is an important area of focus for disciplines such as urban and rural planning and design, and architecture. The skyline plays a vital role in urban spatial identification, urban image presentation, and urban cultural memory. Quantitative skyline assessment allows for quantitative comparison and evaluation of the visual impact on streets, better guiding urban renewal and urban landscape transformation.
[0003] Currently, there are many existing methods for evaluating skylines, including GIS technology, fractal theory, and 3D scene simulation. However, most of these methods are based on existing building shape data and cannot simulate urban skylines from a real human perspective. The skyline outline extraction is not realistic enough, and quantitative descriptive indicators cannot be used to evaluate urban skylines. Summary of the Invention
[0004] The problem to be solved by this invention is to provide a method for evaluating urban skylines based on portable devices and deep learning. This method collects real street view images of cities as the basis for skyline analysis data, extracts the urban skyline outline, and constructs a comprehensive evaluation index system for the skyline to quantitatively assess the urban skyline.
[0005] This invention adopts the following technical solution: a method for evaluating urban skylines based on portable devices and deep learning, comprising the following steps:
[0006] S1. Street view image acquisition based on portable devices: Based on the panoramic shooting device on the portable device, continuous street view shooting is carried out to acquire panoramic street view images with latitude, longitude and height information.
[0007] S2. Spatial positioning and correction of street view image acquisition points: The panoramic street view image is positioned as a spatial layer, and the information on duplicate points, offsets, and elevation errors is corrected.
[0008] S3. Segment the panoramic street view image, identify street view elements and re-stitch: Segment the panoramic street view image into cubes, identify street view elements using the PSPNet method, use transfer learning to improve the accuracy of street view element identification, and restore it to a panoramic street view image stitched after feature recognition.
[0009] S4. City skyline contour extraction: Extract the skyline contour from the panoramic street view image stitched together after feature recognition. Compare the three indicators of interval length, interval coverage and interval intersection to improve the correction of the skyline contour.
[0010] S5. Urban Skyline Scoring and Evaluation: Calculate the inflection degree Q, total length L, vertical height difference H between the highest and lowest points, and horizontal distance difference D between the highest and lowest points of the skyline contour lines. Score the evaluation items and determine the skyline score results.
[0011] The present invention also provides: an electronic device, comprising:
[0012] One or more processors;
[0013] A storage device on which one or more programs are stored;
[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-described urban skyline evaluation methods.
[0015] The present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in any of the above-mentioned urban skyline evaluation methods.
[0016] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:
[0017] 1. This invention uses portable devices to autonomously collect real street scene images of cities. Compared with the traditional method of relying on building vector data for shape analysis, it takes into account more of the surrounding environment of buildings, including elements such as the sky and plants, and provides a more intuitive and realistic analysis of the skyline outline.
[0018] 2. This invention enhances the recognition effect of street scene elements by using deep learning for element recognition and skyline contour extraction, extracts the urban skyline contour line, and constructs a comprehensive evaluation index system for the skyline based on the vectorization of the contour line, using indicators such as tortuosity, total length, height difference, and distance difference to quantitatively evaluate the urban skyline.
[0019] 3. The urban skyline evaluation method of this invention inputs relevant street view data into an artificial intelligence framework. The analysis results in an unsupervised environment are highly aggregated, which reduces the data collection and learning costs, improves the intelligence and accuracy of skyline evaluation, and can provide guidance for urban renewal and landscape transformation. Attached Figure Description
[0020] Figure 1 This is a flowchart of the urban skyline evaluation method of the present invention;
[0021] Figure 2 This is an example of a panoramic street view image and a cube-cropped image from an embodiment of the present invention;
[0022] Figure 3This is a comparison of the recognition results of the present invention PSPNet with those optimized by transfer learning;
[0023] Figure 4 This is a street view panorama and recognition result according to an embodiment of the present invention;
[0024] Figure 5 This is the result of street view recognition and skyline extraction in an embodiment of the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0026] In one embodiment of the present invention, a method for evaluating urban skylines based on portable devices and deep learning is provided, such as... Figure 1 As shown, it includes the following steps:
[0027] The first step is to collect street view images using portable devices.
[0028] In this embodiment, a dedicated panoramic camera is selected, mounted on the moving object and fixed in a fixed position, the shooting angle is adjusted, and the moving object is kept moving at a constant speed so that street scene shooting can be carried out continuously at certain intervals and distances to collect 360° panoramic images with information such as latitude, longitude, and altitude.
[0029] To ensure the quality of street view images, they need to be captured in weather conditions with moderate lighting to avoid overexposure or insufficient image clarity.
[0030] The second step is the spatial positioning and calibration of street view image collection points.
[0031] (1) Extract the latitude and longitude spatial information X and Y of the image acquisition point recorded when the panoramic image was taken from the attribute information of the panoramic street view image, and import it into the spatial database according to the spatial location to form the spatial layer of the image acquisition point.
[0032] (2) Perform spatial topology analysis on the spatial layer of image acquisition points. If image acquisition points that are less than N meters apart are found, they are judged as duplicate points. Only one of them is randomly retained as the image acquisition point for that spatial location.
[0033] (3) Combine the road network vector data to perform route offset analysis on the image acquisition points, that is, calculate the plane vertical distance from the image acquisition point to the nearest road. If the plane vertical distance is greater than M meters, it is considered that the image acquisition point deviates from the acquisition route by more than the normal threshold. It should be an offset acquisition point whose spatial position is incorrectly recorded by the acquisition device. For offset points, they are directly removed from the spatial layer.
[0034] (4) Check the spatial elevation of the remaining image acquisition points. For each image acquisition point, calculate the average spatial elevation of the four image acquisition points in the preceding and following order. If the absolute value of the spatial elevation of the point and the average value is greater than P meters, it is considered that there is an error in the spatial elevation record of the image acquisition point. The average spatial elevation is used to replace the original elevation value, that is, the local K-means interpolation method is used to correct the elevation error.
[0035] In this embodiment, N=1, M=5, and P=10.
[0036] Step 3: Segmentation of panoramic street view images, identification of street view elements, and re-stitching.
[0037] (1) Divide the located and corrected panoramic street view image into cubes, according to six directions: top, bottom, left, right, front, and back. Figure 2 As shown.
[0038] (2) The segmented images are input in batches into PSPNet (Pyramid Scene Parsing Network) for street scene element recognition and analysis.
[0039] In this embodiment, the model is optimized for street view element recognition by using Google's GLDv2 (Google LandmarksDatasets v2) dataset through an autonomous cultivation method.
[0040] The PSPNet feature extraction part is based on the ResNet-101 convolutional neural network, which adds semantic feature extraction, identifies a wider range of image information through each pixel, and uses dilated convolution to make the convolution kernel cover a larger area. The input feature map is denoted as x, the output feature extraction is denoted as R, and the collected features are denoted as H, W, and D, representing the height, width, and depth of the feature, respectively.
[0041] When extracting features from the output, the feature map x∈R H×W×D .
[0042] Based on ResNet-101, pyramid pooling is used to perform average pooling at different scales on the feature maps. The pooled feature maps are then upsampled to their original size and concatenated with the original feature maps. Assume P(x) is the entire pooling module, P...l (x) is an average pooling operation with a scale of l×l performed on the feature map input.
[0043] For each pooled sub-region, the average value is calculated, and then x is sampled and enlarged to a certain size, denoted as U-scalarization of x on a scale of s×s. s The sampling of (x) is represented by the entire pooling module as follows:
[0044] P(x)=[x,U H×W (P1(x)),U H×W (P2(x)),...,U H×W (P L (x))]
[0045] Through pooling, P(x)∈R H×W×(D+L) As a new output feature method, it has a pyramid-level feature representation, which is richer in information than simple feature extraction.
[0046] To improve feature recognition accuracy, transfer learning is further utilized on the basis of PSPNet to optimize street scene feature recognition. By labeling the content in datasets such as GLDv2, the recognition results of the general model for the shooting area are improved.
[0047] The specific method is: [The data source domain D] s The content in the middle is used as the source task Ts, and PSPNet is used as the target domain D. t The target task T t In the middle. Assume x i ∈R H×W×D y is the feature input of the i-th sample. i ∈{0,1} is the class label of the i-th sample. It is a source domain dataset such as GLDv2. It is the target domain dataset of PSPNet after pyramid pooling, n s n t These represent the total number of data points in the source domain dataset and the total number of data points in the target domain dataset, respectively, with the index of n consistent with the index of the dataset.
[0048] In the migration process, Z is used as the hypothesis space after the migration, z is set as the migration complexity, and w is... i As the weights of the i-th sample, λ and γ are used to control the influence of the source domain sample input. The convergence loss is determined by F(z,D). The actual optimization scheme in this embodiment is represented on PSPNet as follows:
[0049]
[0050] (3) Compare the results without transfer learning optimization with the results with transfer learning optimization to determine the recognition effect after learning from specific labels and markers, such as Figure 3 As shown.
[0051] (4) The results further optimized through transfer learning are reverse-engineered into a panoramic image using the principle of cube segmentation, forming a panoramic image stitched together after feature recognition, such as... Figure 4 As shown.
[0052] Step 4: City skyline contour extraction. Based on the feature extraction of the street view image, the skyline contour is extracted from the semantically segmented street view image.
[0053] (1) Obtain a preliminary skyline outline and delineate the marker lines by separating the boundary between the sky and non-sky in the identification results.
[0054] (2) Based on transfer learning, the street scene elements such as streetlights, utility poles, power lines, and traffic signs that have been labeled are transferred out and stored as a feature library model.
[0055] (3) Establish a scan line recognition algorithm, scan the part above the marker line into a segment tree and edge table, forming x∈R corresponding to the semantic segmentation part. H×W×D The feature values are compared with the content in the feature library model to correct errors in the skyline outline.
[0056] (4) A comprehensive comparison of the three indicators of scan line interval length, interval coverage and interval intersection is made to improve the correction of the skyline outline.
[0057] The interval length is used to measure the specific segment of the skyline fitted by the segment tree. Then, the number of coverages is used to determine whether the scan results match the basic corrected skyline. Finally, the slope change between the intersection points of the intervals is used to ensure the accuracy of the optimization correction results.
[0058] Based on the segment tree formed by the scan lines, let... Let C represent the length of the k-th interval. k Let M represent the number of times the k-th interval is covered. Then the length M of the interval with values is expressed as:
[0059]
[0060] The length of the interval also needs to be determined based on the specific number of times the interval is covered to determine whether it fits the feature index library. If it does not fit, it means that further adjustments are needed on top of the basic adjustments.
[0061] Let e k Let y represent the k-th edge. k Let x represent the y-coordinate of the k-th vertex. k Let E(y) represent the x-coordinate of the k-th vertex.k ) represents the set of edges that intersect with the scan line, inout(e k The ') indicates whether it's an incoming or outgoing edge, and is compared with the detection results from the existing model feature library. This confirms whether the skyline matches the segment tree and determines if repair is needed. The coverage count is expressed as:
[0062]
[0063] After comparing the coverage count with the model feature library, the intersection of the intervals is used for the final judgment, through x. k With y k The points formed by these points create an intersection table. Assuming the slope between the intersections is h, the slope is calculated as follows:
[0064] h=(y k+1 -y k ) / (x k+1 -x k )
[0065] The overall slope change of the intersection point table is compared to the slope of the skyline segment (i.e., the extracted features are the same). This determines whether to cover it with colored scan lines or retain the original state. If they match, the original state is retained; otherwise, colored scan lines are used to cover it. Finally, the optimized and corrected skyline outline is extracted. Figure 5 As shown.
[0066] Step 5: Scoring and evaluating the city skyline.
[0067] The extracted skyline contour is vectorized into a continuous polyline. The inflection degree Q, total length L, vertical height difference H between the highest and lowest points, and horizontal distance difference D between the highest and lowest points are calculated as evaluation criteria for professional knowledge. The formula for calculating the inflection degree Q is as follows:
[0068]
[0069] In the formula, ΔH k Let ΔL be the vertical height difference between the k-th pair of adjacent local highest / lowest points. k denoted as the horizontal distance difference between the k-th pair of adjacent local highest / lowest points.
[0070] An online user evaluation platform was built, providing functions for displaying the skyline and filling in scoring items. Users can evaluate and score the four evaluation items of the skyline one by one based on various calculation indicators of the skyline on the evaluation platform, and submit the evaluation results to the backend.
[0071] The rating platform analyzes and statistically analyzes the ratings from all users, automatically generating a scoring and statistical form for each skyline.
[0072] For each evaluation item of the skyline, a comprehensive evaluation index for the skyline outline is calculated using the following formula:
[0073] S = a * Q S +b*L S +c*H S +d*D S
[0074] Where S is the evaluation index, Q S Score L for tortuosity S Score the total length of the broken line, H S Rate the height difference, D S Scoring is assigned to the distance difference; a, b, c, and d are weighting factors, and their sum is 1.
[0075] Specifically, Q S L S H S D S Scoring can be obtained through expert scoring methods.
[0076] After all the comprehensive evaluation indicators for the skylines have been calculated, the skylines are ranked from highest to lowest according to the indicators. The higher the indicator, the better the skyline is considered to be in the environment of the data collection point.
[0077] It should be noted that the comprehensive evaluation index of the skyline outline calculated in this embodiment can be used to assess the urban skyline, which helps to improve the objectivity and scientific nature of relevant departments in urban landscape transformation.
[0078] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered within the scope of protection of the present invention.
Claims
1. A method for evaluating urban skylines based on portable devices and deep learning, characterized in that, Includes the following steps: S1. Street view image acquisition based on portable devices: Based on the panoramic shooting device on the portable device, continuous street view shooting is carried out to acquire panoramic street view images with latitude, longitude and height information. S2. Spatial positioning and correction of street view image acquisition points: The panoramic street view image is positioned as a spatial layer, and the information on duplicate points, offsets, and elevation errors is corrected. S3. Segment the panoramic street view image, identify street view elements and re-stitch: Segment the panoramic street view image into cubes, identify street view elements using the PSPNet method, use transfer learning to improve the accuracy of street view element identification, and restore it to a panoramic street view image stitched after feature recognition. S4. City skyline contour extraction: Extract the skyline contour from the panoramic street view image stitched together after feature recognition. Compare the three indicators of interval length, interval coverage and interval intersection to improve the correction of the skyline contour. The correction of the skyline contour uses an interval length metric line segment tree to fit specific segments of the skyline. Combined with the number of coverage iterations, it determines whether the scan results match the base corrected skyline. The accuracy of the optimized correction is ensured by varying the slope between interval intersections, as detailed below: S4.4.
1. Based on the segment tree formed by the scan lines, let... Indicates the first The length of each interval Indicates the first The length of the interval containing a value is determined by the number of times the interval is covered. Represented as: ; S4.4.
2. Determine whether the interval length fits the feature index library based on the interval coverage count. The coverage count is expressed as: ; set up Indicates the first Strip edge, Indicates the first vertices coordinate, Indicates the first vertices coordinate, This represents the set of edges that intersect with the scan line. Indicate whether it is an incoming or outgoing edge, compare it with the detection results of the existing model feature library, and confirm whether the skyline is consistent with the segment tree. If they are inconsistent, continue to correct it. S4.4.3 After comparing the coverage count with the model feature library, the intersection of the intervals is used for judgment. and The pairs of points form an intersection table, assuming the slope between the intersections is... The slope is calculated as follows: h= - ; Compare the overall slope change of the intersection table with the slope of the skyline segment. If they are consistent, retain the original state. If they are inconsistent, use colored scan lines to cover and extract the optimized and corrected skyline outline. S5. Urban Skyline Scoring and Evaluation: Calculate the inflection degree Q, total length L, vertical height difference H between the highest and lowest points, and horizontal distance difference D between the highest and lowest points of the skyline contour lines. Score the evaluation items and determine the skyline score results.
2. The urban skyline evaluation method according to claim 1, characterized in that, Step S2, the spatial positioning and correction of street view image acquisition points, includes the following sub-steps: S2.1 Extract the latitude and longitude spatial location information X and Y of the image acquisition point recorded during shooting from the attribute information of the panoramic street view image, and import it into the spatial database according to the spatial location to form a spatial layer of image acquisition point; S2.2 Perform spatial topology analysis on the spatial layer of image acquisition points, identify image acquisition points that are less than N meters apart as duplicate points, and randomly retain one of them as the image acquisition point for that spatial location. S2.
3. Combine road network vector data to perform route offset analysis on image acquisition points, calculate the vertical distance from the image acquisition point to the nearest road. If the vertical distance is greater than M meters, the image acquisition point deviates from the acquisition route by more than the normal threshold. It is an offset acquisition point with incorrect spatial location. The offset acquisition point is removed from the spatial layer. S2.4 Check the spatial elevation of the remaining image acquisition points and use the local K-means interpolation method to correct the elevation error: For each image acquisition point, calculate the average spatial elevation of the four image acquisition points in the preceding and following order. If the absolute value of the spatial elevation of the image acquisition point and the average spatial elevation is greater than P meters, then the spatial elevation record of the image acquisition point has an error, and the average spatial elevation is used to replace the spatial elevation value of the image acquisition point.
3. The urban skyline evaluation method according to claim 1, characterized in that, Step S3, the spatial positioning and correction of street view image acquisition points, includes the following sub-steps: S3.
1. Divide the positioned and calibrated panoramic street view image into cubes, according to six directions: top, bottom, left, right, front, and back. S3.
2. Input the segmented images in batches into the pyramid scene analysis neural network for street scene element recognition and analysis. Through transfer learning, optimize the recognition of street scene elements. S3.3 Compare the results without transfer learning optimization with the results with transfer learning optimization to determine the recognition effect after learning from specific labels and markers; S3.
4. The results optimized through transfer learning are reverse-engineered into panoramic street view images using the principle of cube segmentation, forming panoramic street view images stitched together after feature recognition.
4. The urban skyline evaluation method according to claim 3, characterized in that, In step S3.2, the pyramid scene analysis neural network, denoted as PSPNet, is used to perform street scene element recognition and analysis. The method is as follows: Feature extraction is performed based on the ResNet-101 convolutional neural network. The feature map input is x. When extracting features: ; Feature extraction is denoted as The collected features are , , , representing the height, width, and depth of the feature, respectively; The pyramid pooling method is used to perform average pooling at different scales on the feature map. The pooled feature map is then upsampled to its original size and concatenated with the original feature map. This represents the entire pooling module. This indicates that the feature map input is scaled by... The average pooling operation calculates the average value for each pooled sub-region, and then... Top conduct Sampling, Represented as: ; Where L represents the depth of pooling stacking; through pooling This is a feature representation with a pyramid-like number of levels.
5. The urban skyline evaluation method according to claim 4, characterized in that, In step S3.2, transfer learning is used to optimize street scene element recognition. The method is as follows: Data source domain The content in the source task Implantation into the PSPNet target domain The target task ; ; ; in, It is the target domain dataset after PSPNet pyramid pooling. It is the first Each sample feature is input, It is the first The category label of each sample, , These represent the total number of data points in the source domain dataset and the total number of data points in the target domain dataset, respectively, with the index of n matching the dataset index. During the migration, As the hypothesis space after migration For samples in the hypothesis space, Let the transfer complexity be denoted as follows, the optimized method for street scene feature recognition on PSPNet is expressed as follows: ; in, As the first The weights of each sample, λ and The loss used to control the influence of source domain sample inputs is used to determine convergence. .
6. The urban skyline evaluation method according to claim 2, characterized in that, In step S4, the skyline outline is extracted from the stitched panoramic street view image after feature recognition, as follows: S4.
1. By segmenting and identifying the boundary between the sky and non-sky in the street scene elements, a preliminary skyline outline is obtained and a marker line is drawn. S4.2 Based on transfer learning, the labeled street scene elements such as streetlights, utility poles, power lines, and traffic signs are transferred out and stored as a feature library model. S4.3 Establish a scan line recognition algorithm, scanning the portion above the marker line into a segment tree and edge table, forming a structure corresponding to the semantic segmentation portion. The feature values are compared with the content in the feature library model to correct errors in the skyline outline; S4.
4. A comprehensive comparison of the three indicators of scan line interval length, interval coverage and interval intersection is performed to improve the correction of the skyline outline.
7. The urban skyline evaluation method according to claim 6, characterized in that, Step S5, the scoring and evaluation of the city skyline, includes the following sub-steps: S5.1 Vectorize the extracted skyline contour into a continuous polyline, and calculate the polyline's tortuosity Q, total length L, vertical height difference H between the highest and lowest points, and horizontal distance difference D between the highest and lowest points, as scoring evaluation items; The formula for calculating the tortuosity Q is as follows: ; In the formula, For the first For the vertical height difference between adjacent local highest / lowest points, For the first The difference in horizontal distance between adjacent local highest / lowest points; S5.2 Build an online user evaluation platform that provides the functions of displaying the skyline and filling in the scoring items. On the evaluation platform, users can evaluate and score the four evaluation items of the skyline one by one based on the various calculation indicators of the skyline, and submit the evaluation results to the backend. S5.3 The online user rating platform analyzes and statistically analyzes the rating results of all users and automatically generates a rating statistics form for each skyline. S5.
4. For each evaluation item of the skyline, score it and calculate the comprehensive evaluation index of the skyline outline. The formula is as follows: S=a*Q S +b*L S +c*H S +d*D S ; Where S is the evaluation index, Q S Score L for tortuosity S Score the total length of the broken line, H S Rate the height difference, D S Scoring is done on the distance difference; a, b, c, and d are weighting factors, and a+b+c+d=1; S5.5 After all the comprehensive evaluation indicators of the skyline are calculated, the skylines are sorted from high to low according to the indicators. The higher the indicator, the better the skyline of the environment of the collection point.
8. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the urban skyline evaluation method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Panorama image acquisition system, and apparatus and method for measuring skyline based on the same
CN105516584A
A digital city skyline extraction method
CN109285177A
