Comprehensive landscape representation method and device based on street images and storage medium

Abstract

The invention discloses a comprehensive landscape representation method and device based on street images and a storage medium, street scene features are extracted through artificial intelligence, and then a street scene image and a geographic space are mapped so as to realize comprehensive landscape representation of the street scene image. According to the method, the streetscape image is used, so that the microstructure and landscape characteristics in a city can be more intuitively reflected. According to the method, the limitation that landscape representation is carried out only by depending on a single street image in the prior art is broken through, and the urban landscape characteristics can be represented more comprehensively and accurately through comprehensive processing of multiple street images.

Description

Technical Field

[0001] The present invention relates to the fields of information technology and urban geographic technology, and in particular to a comprehensive landscape representation method, device and storage medium based on street images. Background Art

[0002] Urban landscape research is an important part of urban planning, environmental protection, and smart city construction. With the acceleration of global urbanization, the study of urban landscape not only helps to understand and analyze the current development status of cities, but also provides a scientific basis for future urban planning and policy making. By studying urban landscape, we can reveal information on the spatial structure, land use pattern, ecological environment, and social and economic activities of cities. These research results can help urban managers formulate more reasonable and sustainable development strategies and improve the quality of life of urban residents. The main technical means for studying urban landscape at present is remote sensing imaging technology. Remote sensing imaging technology obtains surface image data through satellites or aircraft, and can capture the overall picture of the city in a large range and quickly. This method can obtain high-resolution image data for accurate spatial analysis and change monitoring. Using remote sensing imaging technology, researchers can analyze information on urban land use changes, green space distribution, building density, etc. However, remote sensing imaging technology also has certain limitations, such as the difficulty in obtaining detailed information inside the city and the limitation of its application under complex weather conditions.

[0003] In addition to remote sensing imaging technology, existing technologies also use street images to study urban scenes. Street images are images taken on the ground by street view cars or personal devices, which can provide detailed visual information inside the city, such as building facades, street facilities, vegetation conditions, etc. These image data can help researchers better understand the microstructure and landscape characteristics of the city. Existing technologies are still at the stage of using a single street view image to represent a single scene, but the urban landscape is often a collection of multiple different urban scenes, and the number of urban scenes varies greatly in different spatial units at different spatial scales. How to comprehensively express the urban landscape based on complex multi-city scenes remains a difficult problem to solve. Summary of the invention

[0004] In order to remedy the above-mentioned defects, the present application proposes a comprehensive landscape representation method, device and storage medium based on street images.

[0005] The method and technical solution is: a comprehensive landscape representation method based on street images, including: Acquire a street image set of the area to be landscaped, wherein each image in the street image set includes a geographical location where the image was taken; Get the vector geographic space range of the area to be landscaped; The vector geographic space range and the street image set are spatially connected based on the geographic location to obtain the spatial mapping relationship between the vector geographic space range and the street image set; A street view image set located within the vector geographic space is screened and extracted from the street image set according to the spatial mapping relationship; Using a pre-trained scene feature extractor to perform feature extraction on each image in the street view image set, respectively, to obtain an image embedding feature of each image, wherein the image embedding feature is a feature representation of the image scene; Calculate the average value and standard deviation of the embedded features of all images in the street view image set to obtain the average embedded features and the standard deviation embedded features respectively, and concatenate the average embedded features and the standard deviation embedded features to obtain the concatenated features; The splicing features are used as a comprehensive landscape representation of the area to be landscape represented.

[0006] Preferably, the process of screening and extracting a street view image set within the vector geographic space range from the street image set according to the spatial mapping relationship also includes dividing the vector geographic space range into multiple space units according to geographic features, and screening and extracting sub-street view image sets corresponding to each space unit from the street view image set according to the spatial mapping relationship, and the multiple sub-street view image sets constitute the street view image set.

[0007] Preferably, a pre-trained scene feature extractor is used to perform feature extraction on each image in the street view image set to obtain image embedding features of each image. The process includes: a pre-trained scene feature extractor is used to perform feature extraction on each image in each sub-street image set to obtain image embedding features of each image in the corresponding sub-street image set, and the corresponding image embedding features are used as feature representations of the spatial unit image scene corresponding to the sub-street image set.

[0008] Preferably, the average value and standard deviation of the embedded features of all images in the street view image set are calculated to obtain the average embedded features and the standard deviation embedded features respectively, and the average embedded features and the standard deviation embedded features are spliced. The process of obtaining the spliced ​​features includes: based on the sub-street view image set corresponding to each spatial unit, the average value and the standard deviation of the image embedded features in the sub-street view image set are calculated to obtain the average embedded features and the standard deviation embedded features of the sub-street view image set, and the average embedded features and the standard deviation embedded features are spliced ​​to obtain the spliced ​​features of the corresponding sub-street view image set.

[0009] Preferably, the process of using the splicing features as the comprehensive landscape representation of the area to be landscape represented includes using the splicing features of the sub-street view image set as the comprehensive landscape representation of the corresponding spatial unit, and using the comprehensive landscape representation of all spatial units as the comprehensive landscape representation of the area to be landscape represented.

[0010] Preferably, before using a pre-trained scene feature extractor to perform feature extraction on each image in the street view image set, each image in the street view image set is also pre-processed to unify pixels and sizes.

[0011] Preferably, the preprocessing process includes unifying the pixel size of each image.

[0012] The present application also proposes an electronic device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements any of the methods described above when executing the computer program.

[0013] The present application also proposes a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described above.

[0014] The present invention can more intuitively reflect the microstructure and landscape features of the city by using street view images. The present invention breaks through the limitation of the existing technology that only relies on a single street image for landscape representation. By comprehensively processing multiple street images, the urban landscape features can be more comprehensively and accurately represented. The present invention uses a pre-trained deep learning model to extract features from street images, which can accurately capture the important features in each image, and by calculating the average and standard deviation of these features, a high-precision landscape representation of spatial units can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 The present invention is a schematic flow chart of an embodiment of the method of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0018] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0019] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0020] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0021] Combination Figure 1 , an implementation of the present application. A comprehensive landscape representation method based on street images, comprising:

[0022] A street image set of an area to be represented by landscape is obtained, wherein each image in the street image set contains the geographical location where the image was taken; a vector geographic space range of the area to be represented by landscape is obtained; the vector geographic space range and the street image set are spatially connected based on the geographical location to obtain a spatial mapping relationship between the vector geographic space range and the street image set; a street view image set within the vector geographic space range is screened and extracted from the street image set according to the spatial mapping relationship; a pre-trained scene feature extractor is used to perform feature extraction on each image in the street view image set to obtain an image embedding feature of each image, wherein the image embedding feature is a feature representation of the image scene; an average value and a standard deviation of the embedded features of all images in the street view image set are calculated to obtain an average value embedded feature and a standard deviation embedded feature, respectively, and the average value embedded feature and the standard deviation embedded feature are spliced ​​to obtain a spliced ​​feature; and the spliced ​​feature is used as a comprehensive landscape representation of the area to be represented by landscape.

[0023] Based on the above embodiment, further, the process of screening and extracting a street view image set located within the vector geographic space range from the street image set according to the spatial mapping relationship also includes dividing the vector geographic space range into multiple space units according to geographic features, and screening and extracting sub-street view image sets corresponding to each space unit from the street view image set according to the spatial mapping relationship, and the multiple sub-street view image sets constitute the street view image set.

[0024] Based on one or more of the above embodiments, further, a pre-trained scene feature extractor is used to perform feature extraction on each image in the street view image set to obtain an image embedding feature of each image, which includes: a pre-trained scene feature extractor is used to perform feature extraction on each image in each sub-street image set to obtain an image embedding feature of each image in the corresponding sub-street image set, and the corresponding image embedding feature is used as a feature representation of the spatial unit image scene corresponding to the sub-street image set.

[0025] On the basis of one or more of the above embodiments, further, the average value and standard deviation of the embedded features of all images in the street view image set are calculated to obtain the average value embedded features and the standard deviation embedded features respectively, and the average value embedded features and the standard deviation embedded features are spliced ​​to obtain the spliced ​​features, and the process of obtaining the spliced ​​features includes: based on the sub-street view image set corresponding to each spatial unit, the average value and standard deviation of the image embedded features in the sub-street view image set are calculated to obtain the average value embedded features and the standard deviation embedded features of the sub-street view image set, and the average value embedded features and the standard deviation embedded features are spliced ​​to obtain the spliced ​​features of the corresponding sub-street view image set.

[0026] Based on one or more of the above embodiments, further, the process of using the stitching features as the comprehensive landscape representation of the area to be landscape represented includes: using the stitching features of the sub-street view image set as the comprehensive landscape representation of the corresponding spatial unit, and using the comprehensive landscape representation of all spatial units as the comprehensive landscape representation of the area to be landscape represented.

[0027] On the basis of one or more of the above embodiments, further, the present application further uses cities as study areas, and expresses the comprehensive landscape of their street scenes for illustration. This embodiment selects 36 Chinese cities with different economic development levels and geographical locations as study areas, including obtaining the street view image set of Tencent Maps of 36 Chinese cities, which contains a total of 8,189,058 street view images, wherein each image contains the longitude and latitude information of its shooting, that is, its shooting geographical location information; obtains the administrative district vector space data of the 36 cities, the administrative district vector space data is the vector geographic space range data, in the rectangular coordinate system, each coordinate axis data is used to represent the data of the position and shape of the map graphics or geographical entities, and in this embodiment, the vector geographic space range data of the above 36 cities is obtained according to the administrative district vector space data of the 36 cities in the form of software input. Subsequently, this embodiment spatially connects the vector geographic space range with the street image set based on the geographical location, that is, spatially connects the longitude and latitude information of the 36 cities with the longitude and latitude geographical information of the street image set taken, so that they are mapped together according to the geographical location. This embodiment also includes a troubleshooting process during the mapping process, which is to query whether the longitude and latitude information of 36 cities correspond to the longitude and latitude geographic information of the street image set taken. If there is no corresponding image in a certain city or a certain area of ​​a certain city, the area will be deleted, and the area with a mapping relationship will be retained. Subsequently, the pre-trained scene feature extractor is used to extract features for these street view images for which spatial mapping relationships have been established. In this example, the ResNet-18 model pre-trained based on the Places365 data set is used for feature extraction, and the image embedding feature of each image is 512 dimensions, which is the feature representation of each scene. Afterwards, the mean and standard deviation of these image embedding features are calculated to obtain the mean embedding feature and standard deviation embedding feature of each 512 dimension respectively. The mean embedding feature and the standard deviation embedding feature are concatenated to obtain a 1024-dimensional embedding feature. This deep feature is the comprehensive landscape feature representation. In this embodiment, in order to improve the operation speed, the 36 cities can be further screened, mapped and expressed by dividing the spatial units into sub-street view image sets. For example, based on the kilometer grid as the basic spatial unit, the administrative district vector spatial data of each city is divided into multiple kilometer grids, and all the kilometer grids of each city are converted into one spatial unit. Similarly, according to the spatial mapping relationship, the sub-street view image set corresponding to the spatial unit where the city is located is screened and extracted from the street view image set, and the images of the sub-street view image set are further mapped to each basic spatial unit. If there is no street view image in the kilometer grid, the kilometer grid is not included in the subsequent analysis process. In this way, each kilometer grid corresponds to a number of street view images. Through this method, the overall area to be expressed is split into separate kilometer grids for expression, which reduces the amount of calculation each time and improves the calculation efficiency.Similarly, in the process of calculating the comprehensive landscape feature representation, the basic spatial unit is still used. According to the street images corresponding to each kilometer grid, the average value and standard deviation of the embedded features of these images are calculated, and the average value embedded features and standard deviation embedded features of each kilometer grid are obtained respectively. The average value embedded features and the standard deviation embedded features are spliced ​​to obtain 1024-dimensional embedded features. The depth feature is the comprehensive landscape feature representation of the kilometer grid. In view of the defect that the existing remote sensing imaging technology mainly provides a bird's-eye view and it is difficult to obtain detailed information inside the city, this embodiment uses street view images from the perspective of pedestrians to more intuitively reflect the microstructure and landscape features inside the city, such as building facades, street facilities and vegetation conditions. The present invention breaks through the limitation of the prior art that only relies on a single street image for landscape representation. Through the comprehensive processing of multiple street images, the urban landscape features can be more comprehensively and accurately represented. This embodiment uses a pre-trained deep learning model to extract features from street images, which can accurately capture the important features in each image, and by calculating the average value and standard deviation of these features, a high-precision landscape representation of the spatial unit is achieved.

[0028] Based on one or more of the above embodiments, further, before using a pre-trained scene feature extractor to perform feature extraction on each image in the street view image set, each image in the street view image set is pre-processed to unify pixels and sizes.

[0029] Based on one or more of the above embodiments, further, the preprocessing process includes unifying the pixel size of each image. In this embodiment, since the images used in this application may come from multiple shooting devices, and there are pixel differences between each shooting device, in order to facilitate the neural network to extract the features of the image, the pixels of the original image of this application are unified.

[0030] An electronic device comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the method described in any one of the above embodiments when executing the computer program.

[0031] A computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a method as described in any one of the above embodiments.

[0032] The above is a detailed introduction to the landscape representation method proposed in the present invention. The above examples illustrate the principle and implementation method of the present invention, but are only used to help understand the method and its core idea of ​​the present invention. At the same time, according to different application scenarios, there will be changes in the specific implementation methods. In summary, the above examples should not be understood as limiting the present invention.

Claims

1. A comprehensive landscape representation method based on street images, characterized in that: include: Acquire a street image set of the area to be landscaped, wherein each image in the street image set includes a geographical location where the image was taken; Get the vector geographic space range of the area to be landscaped; The vector geographic space range and the street image set are spatially connected based on the geographic location to obtain the spatial mapping relationship between the vector geographic space range and the street image set; A street view image set located within the vector geographic space is screened and extracted from the street image set according to the spatial mapping relationship; Using a pre-trained scene feature extractor to perform feature extraction on each image in the street view image set, respectively, to obtain an image embedding feature of each image, wherein the image embedding feature is a feature representation of the image scene; Calculate the average value and standard deviation of the embedded features of all images in the street view image set to obtain the average embedded features and the standard deviation embedded features respectively, and concatenate the average embedded features and the standard deviation embedded features to obtain the concatenated features; The splicing features are used as a comprehensive landscape representation of the area to be landscape represented.

2. The method according to claim 1, characterized in that: The process of selecting and extracting a street view image set within the vector geographic space range from the street image set according to the spatial mapping relationship also includes dividing the vector geographic space range into a plurality of spatial units according to geographic features, selecting and extracting a sub-street view image set corresponding to each spatial unit from the street view image set according to the spatial mapping relationship, and the plurality of sub-street view image sets constitute the street view image set.

3. The method according to claim 2, characterized in that A pre-trained scene feature extractor is used to perform feature extraction on each image in the street view image set to obtain image embedding features of each image. The process includes: a pre-trained scene feature extractor is used to perform feature extraction on each image in each sub-street image set to obtain image embedding features of each image in the corresponding sub-street image set, and the corresponding image embedding features are used as feature representations of the spatial unit image scene corresponding to the sub-street image set.

4. The method according to claim 3, characterized in that: The average value and standard deviation of the embedded features of all images in the street view image set are calculated to obtain the average embedded features and the standard deviation embedded features respectively, and the average embedded features and the standard deviation embedded features are spliced ​​to obtain the spliced ​​features. The process includes: based on the sub-street view image set corresponding to each spatial unit, the average value and the standard deviation of the image embedded features in the sub-street view image set are calculated to obtain the average embedded features and the standard deviation embedded features of the sub-street view image set, and the average embedded features and the standard deviation embedded features are spliced ​​to obtain the spliced ​​features of the corresponding sub-street view image set.

5. The method according to claim 4, characterized in that The process of using the splicing features as the comprehensive landscape representation of the area to be represented includes using the splicing features of the sub-street view image set as the comprehensive landscape representation of the corresponding space unit, and using the comprehensive landscape representation of all space units as the comprehensive landscape representation of the area to be represented.

6. The method according to claim 1 or 3, characterized in that: Before using a pre-trained scene feature extractor to perform feature extraction on each image in the street view image set, each image in the street view image set is also pre-processed to unify pixels and sizes.

7. The method according to claim 6, characterized in that The preprocessing process includes unifying the pixel size of each image.

8. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Images