Urban street scene representation method based on multilevel semantic tag enhancement

Abstract

The invention relates to the technical field of computer vision and artificial intelligence, in particular to an urban street scene representation method based on multilevel semantic tag enhancement, which comprises the following steps: acquiring an urban street scene image, and constructing a street physical semantic feature model based on visual elements; constructing a street space semantic feature model based on the visual elements; constructing a street function semantic feature model based on the interest points; and constructing a street scene representation model based on the graph convolutional neural network. According to the method, the problems of single semantic granularity and single semantic type in urban street scene representation in an existing method are solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method for representing urban street scenes based on multi-level semantic label enhancement. Background Technology

[0002] Urban street scene representation is the process of digitally, structurally, and semantically describing and expressing the physical environment and dynamic characteristics of urban streets, transforming complex objects and their interactions in the physical world into information that computers can recognize and reason about.

[0003] Urban street scene representation is a core foundational task for analyzing urban landscape, function, and layout, and for depicting downstream tasks such as residents' travel, life, and emotions.

[0004] Existing technologies mainly achieve urban street scene representation through the following steps: street scene visual element identification and statistics, identifying visual element types by semantic segmentation of street scene images of street sampling points, and counting their pixel count and proportion; designing a loss function based on the similarity of the distribution of visual element features of street sampling points, and constructing an urban street scene representation model based on deep neural networks.

[0005] However, existing methods have significant technical drawbacks: 1. The semantic level or granularity of visual elements in street view images is difficult to determine. If the granularity is too coarse, it will easily lead to insufficient representation of scene details and element diversity, and loss of key information. If the granularity is too fine, it will easily introduce noise, increase the difficulty of element recognition, and ignore the combination pattern and overall structure between elements. 2. It mainly considers the physical semantics of visual elements in street scenes, but ignores the representation of potential human activity patterns caused by the similarity between their spatial semantics and functional semantics, making it difficult to achieve a comprehensive perception and accurate understanding of urban street scenes. Summary of the Invention

[0006] To address the shortcomings of existing methods, this invention solves the problems of limited semantic granularity and limited semantic types in the representation of urban street scenes.

[0007] The technical solution adopted in this invention is: a method for representing urban street scenes based on multi-level semantic tag enhancement, comprising the following steps: Step 1: Acquire urban street view images and construct a street physical semantic feature model based on visual elements; As a preferred embodiment of the present invention, step one specifically includes: Step 11: Perform distance segmentation on the urban road network, obtain street sampling points, and collect street view images from different angles along the road's forward direction; Step 12: Construct the physical semantic feature vector of visual elements in street sampling point images. , , Representing the physical semantic level Down, sampling direction The pixel ratio feature vectors of each visual element in the corresponding street view image; As a preferred embodiment of the present invention It utilizes image semantic segmentation algorithms to segment physical semantics at various levels. The system identifies the types of visual elements in street view images and calculates the pixel ratio of each visual element.

[0008] Step 13: Generate image visual feature vectors with enhanced physical semantic labels. , Physical semantic tag tree for pre-defined street scene visual elements Word embedding is performed to obtain feature vectors of physical semantic labels at each level; Step 14, for By fusing the samples, we can obtain the physical semantic feature vectors of the street sampling points. , This indicates the hierarchical size of the physical semantic tag tree.

[0009] Step 2: Construct a street space semantic feature model based on visual elements; In a preferred embodiment of the present invention, step two specifically includes: Step 21: Construct spatial semantic feature vectors of visual elements in street sampling point images. ; In physical semantics Down, sampling direction The spatial adjacency matrix of various visual elements in the corresponding street view image; Step 22, and Fusion generates spatially semantically enhanced image visual feature vectors ; Step 23, for By fusing the data, we obtain multi-level spatial semantic feature vectors of street sampling points. .

[0010] Step 3: Construct a street function semantic feature model based on points of interest; In a preferred embodiment of the present invention, step three specifically includes: Step 31: Using the street sampling point as the center, collect points of interest within the preset buffer zone; Step 32: Construct functional semantic feature vectors for street sampling points based on interest point functional semantic label vectors. ; Representing functional semantic hierarchy Next The proportion of interest points for each functional type Representing functional semantic hierarchy The scale of the Point of Interest (POI) feature type; Step 33, for By fusing the data, we obtain multi-level functional semantic feature vectors of street sampling points. , This indicates the hierarchical size of the semantic tag tree representing points of interest. This represents the eigenvector concatenation function.

[0011] Step 4: Construct a representation model of urban street scenes based on graph convolutional neural networks; In a preferred embodiment of the present invention, step four specifically includes: Step 41, using Calculate the similarity of multi-level functional semantics between two sampling points in the sample; In a preferred embodiment of the present invention, the formula for similarity is: , ; in, Indicates a sample, Indicates the number of samples. Indicates Jenson's Shannon divergence. Indicates sampling point pair Multi-level functional semantic feature vector combination.

[0012] Step 42: Set the functional semantic similarity threshold; Step 43, according to Design a loss function based on the road network distance between sampling point pairs. , Indicates the number of positive samples. Indicates the number of negative samples. This represents the road network distance between street sampling point pairs in the positive sample. This represents the road network distance between street sampling point pairs in the negative sample; Step 44: Construct the graph structure , This represents the set of nodes formed by street sampling points. This represents the set of edges formed by the topological structure of the street sampling points. This represents a set of node attributes composed of feature vectors from street sampling points that integrate multi-level physical and spatial semantics; based on A city street scene representation model is trained using a graph convolutional neural network, and the city street scene representation vector is output.

[0013] In a preferred embodiment of the present invention, the formula for the urban street scene representation vector is: ; in, This represents a graph neural network model. Indicates the first Each street sampling point is used to construct a feature vector by fusing multi-level physical semantics and spatial semantics. This indicates the number of street sampling points.

[0014] As a preferred embodiment of the present invention, a city street scene representation system based on multi-level semantic tag enhancement includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement a city street scene representation method based on multi-level semantic tag enhancement.

[0015] As a preferred embodiment of the present invention, a computer-readable medium storing computer program code implements a method for representing urban street scenes based on multi-level semantic tag enhancement when executed by a processor.

[0016] The beneficial effects of this invention are: 1. This invention creatively constructs a visual element representation model for urban street scenes that integrates multi-level semantic tags, solving the problem that the semantic level or granularity of visual elements is difficult to determine in traditional methods. It achieves accurate understanding and effective representation of street scenes, and can significantly improve the comprehensiveness and accuracy of downstream urban application decisions. 2. This invention creatively constructs an unsupervised deep learning model for representing urban street scenes by integrating multiple types of semantic tags. By using multi-level functional semantics of interest points, it solves the problem that traditional methods only consider explicit physical environment features while ignoring potential human activity pattern features. It achieves deep integration of physical world information and social activity information, which can significantly improve the effectiveness and rationality of downstream urban application decisions. Attached Figure Description

[0017] Figure 1 This is a flowchart of a method for representing urban street scenes using multi-level semantic tag enhancement. Figure 2 This is a schematic diagram of multi-level physical semantic feature modeling of visual elements in urban street scenes; Figure 3 This is a schematic diagram of a multi-level physical semantic tag tree for visual elements in an urban street scene; Figure 4 This is a schematic diagram of multi-level spatial semantic feature modeling of visual elements in urban street scenes; Figure 5 This is a schematic diagram of multi-level functional semantic feature modeling of urban street scenes based on points of interest. Figure 6 This is a schematic diagram of a multi-level functional semantic tag tree for points of interest in urban streets; Figure 7 This is a diagram showing the results of a comparative experiment on urban road traffic flow prediction tasks. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0019] like Figure 1 As shown, a method for representing urban street scenes based on multi-level semantic label enhancement includes the following steps: Step 1: Acquire urban street view images and construct a multi-level physical semantic feature model based on the visual elements of urban street view (street scene); like Figure 2 Step 11: According to the actual application requirements, the urban road network data of the study area is segmented by distance (for example, at 50-meter intervals), street sampling points are obtained, and street view image data are collected in four directions: 0°, 90°, 180°, and 270° clockwise along the road direction. Step 12: Based on the preset multi-level physical semantic tag tree of street scene visual elements Using image semantic segmentation algorithms (such as DeepLabV3+ model and UPSNet model), physical semantics at various levels are analyzed. The system identifies the types of visual elements in street scene images, counts their corresponding pixel counts and proportions, and constructs physical semantic feature vectors for these visual elements. The formula is: (1) (2) in, This represents the attention mechanism; , , , The azimuth angles representing the direction of street view image acquisition are 0°, 90°, 180°, and 270°, respectively. Representing the physical semantic level Below, the street sampling points are in the direction Initial values ​​of the visual element feature vectors; Representing the physical semantic level Down, sampling direction The pixel ratio feature vectors of each visual element in the corresponding street view image.

[0020] Physical semantic tag tree This includes the types of features that the image semantic segmentation algorithm needs to identify. For example, if the label tree includes people, cars, sky, and vegetation, then image semantic segmentation needs to segment pixels of the corresponding types. In the formula, this can be understood as... yes A subset, It contains multiple layers of semantic tags. It refers to a specific layer of labels.

[0021] Step 13: Utilize pre-trained word embedding models (e.g., BERT model, Word2vec model, GloVe model) to generate multi-level physical semantic label trees. Word embedding is performed to obtain feature vectors of physical semantic labels at each level. And its corresponding image visual element feature vector The images are fused to generate image visual feature vectors with enhanced physical semantic labels. The formula is: (3) Figure 3 A schematic diagram of a multi-level physical semantic tag tree for urban street scene visual elements. It contains four layers of physical semantic tags; Step 14: Enhance the image visual element feature vectors of each level of physical semantic labels. By fusing the data, multi-level physical semantic feature vectors of street sampling points are obtained. The formula is: (4) in, This indicates the hierarchical size of the physical semantic tag tree.

[0022] Step 2: Construct a multi-level spatial semantic feature model based on urban street view visual elements; like Figure 4 Step 21: Based on the multi-level physical semantic segmentation results of the street scene images from the street sampling points in Step 1, perform physical semantic segmentation at each level. Below, spatial distribution feature recognition algorithms (e.g., image feature neighbor pixel statistics algorithms) are used to construct spatial semantic feature vectors of image visual features based on physical semantic label feature vectors. The formula is: (5) (6) in, Representing the physical semantic level Below, the street sampling points are in the direction Initial values ​​of the spatial semantic feature vectors of visual elements; In physical semantics Down, sampling direction The spatial adjacency matrix of various visual elements in the corresponding street view image; Step 22: Convert the physical semantic label feature vectors at each level Its corresponding image visual element spatial semantic feature vector The images are fused to generate spatially semantically enhanced image visual feature vectors. The formula is: (7) Step 23: Enhance the image visual element feature vectors at each level of spatial semantics. By fusing the data, multi-level spatial semantic feature vectors of street sampling points are obtained. The formula is: (8) Step 3: Construct a multi-level functional semantic feature model of urban streets based on points of interest; like Figure 5 Step 31: Based on actual application needs, collect point of interest data within a preset buffer range (e.g., with a radius of 500 meters) of each street sampling point as the center, and use it as the basic data source for the functional semantic analysis of street sampling points. Step 32: Based on the preset multi-level functional semantic tag tree of interest points At each functional level Next, using spatial analysis algorithms (such as buffer analysis and kernel density analysis), the types, quantities, and proportions of interest points within the street sampling point buffers are calculated. Based on the interest point functional semantic label vectors, functional semantic feature vectors of street sampling points are constructed. The formula is: (9) in, Representing functional semantic hierarchy Next The proportion of interest points for each functional type Representing functional semantic hierarchy The scale of the Point of Interest (POI) feature type.

[0023] Figure 6 This is a schematic diagram of a multi-level functional semantic tag tree for points of interest on city streets, which includes four layers of functional semantic tags.

[0024] Step 33: Fuse the functional semantic feature vectors of street sampling points at each level to obtain multi-level functional semantic feature vectors of street sampling points. The formula is: (10) in, This indicates the hierarchical size of the semantic tag tree representing points of interest. This represents the eigenvector concatenation function.

[0025] Step 4: Construct a representation model of urban street scenes based on graph convolutional neural networks; Step 41: Construct a sample set representing urban street scenes, and pair any street sampling point within the study area. Treating it as a single sample, based on the multi-level functional semantic feature vector of the sampling points obtained in step three... Calculate the multi-level functional semantic similarity between two sampling points in the sample. The formula is: (11) (12) in, Indicates a sample, Indicates the number of samples. Indicates Jenson's Shannon divergence. Indicates sampling point pair Multi-level functional semantic feature vector combination.

[0026] Step 42: Based on the functional semantic similarity distribution characteristics of the sample set, set a functional semantic similarity threshold (e.g., The formula for dividing positive and negative samples is: (13) in, Indicates a positive sample. Indicates a negative sample. Indicates the threshold for functional semantic similarity; Step 43: Design a loss function based on the functional semantic similarity of the sample set and the road network distance between sampling point pairs. The formula is: (14) in, Indicates the number of positive samples. Indicates the number of negative samples. This represents the road network distance between street sampling point pairs in the positive sample. This represents the road network distance between street sampling point pairs in the negative sample.

[0027] Step 44: Based on the urban road network topology, construct a graph structure for the street sampling point set. ;in, This represents the set of nodes formed by street sampling points. This represents the set of edges formed by the topological structure of the street sampling points. This represents the set of node attributes composed of feature vectors from street sampling points that integrate multi-level physical and spatial semantics; based on a loss function, a graph convolutional neural network is used to train a representation model of urban street scenes. The final output is a representation vector of the city street scene. To support downstream city application decisions, the formula is: (15) (16) in, This represents a graph neural network model. Indicates the first Each street sampling point is used to construct a feature vector by fusing multi-level physical semantics and spatial semantics. This indicates the number of street sampling points.

[0028] To verify the effectiveness of this invention, taking the problem of predicting traffic flow speed on urban roads as an example, a comparative experiment was conducted with models such as decision trees, random forests, multilayer perceptrons, and graph neural networks. The results are as follows: Figure 7 As shown; Specifically, the average speed of road segments is first divided into three levels: low speed, medium speed, and high speed. Then, the street scene representation vector of the road segment sampling points extracted in this invention is used as input, and the average speed level of the road segment is used as output to construct a prediction model based on a multilayer perceptron. The comparison model uses a feature vector composed of the pixel proportion of street scene visual elements as input. Experimental results show that the method of this invention can accurately and effectively understand and represent the characteristics of urban street environment, and significantly improve performance in downstream prediction tasks.

[0029] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for representing urban street scenes based on multi-level semantic label enhancement, characterized in that, The method comprises the following steps: Step one, acquiring urban street view images, and constructing a street physical semantic feature model based on visual elements; Step two, constructing a street space semantic feature model based on visual elements; Step three, constructing a street function semantic feature model based on interest points; Step four, constructing a street scene representation model based on a graph convolutional neural network.

2. The method of claim 1, wherein, Step one specifically comprises: Step 11, performing distance segmentation on the urban road network to obtain street sampling points, and collecting street view images at different angles in the forward direction along the road; Step 12, constructing the street sampling point image visual element physical semantic feature vector , , representing the physical semantic hierarchy Next, the collection direction corresponding to the pixel proportion feature vector of each visual element in the street view image; Step 13, generating a physical semantic label enhanced image visual element feature vector , a preset physical semantic label tree of a street view visual element performing word embedding to obtain a feature vector of each level physical semantic label Step 14, fusion is performed on to obtain the physical semantic feature vector of the street sampling point , represents the hierarchical scale of the physical semantic label tree.

3. The method of claim 2, wherein, Step two specifically comprises: Step 21, constructing a street sampling point image visual element space semantic feature vector ; representing the spatial adjacency relationship matrix of each visual element in the corresponding street view image under the physical semantics under the physical semantics corresponding street view image Step 22, generating with fusing to generate a spatially semantically enhanced image visual element feature vector ; Step 23, fusing to obtain a multi-level spatial semantic feature vector of the street sampling point .​ 4. The method of claim 3, wherein, Step three specifically comprises: Step 31, collecting interest points in a preset buffer area centered on the street sampling points; Step 32, constructing the street sampling point function semantic feature vector ; representing the function semantic hierarchy the proportion of the next function type of the interest point, representing the function semantic hierarchy the scale of the function type of the interest point; Step 33, fusing to obtain a multi-level function semantic feature vector of the street sampling point represents a hierarchical scale of the interest point function semantic label tree, represents a feature vector splicing function.​​ 5. The method of claim 4, wherein, Step four specifically comprises: Step 41, utilizing calculating the similarity of the multi-level functional semantics of two sampling points in the sample; Step 42, setting a function semantic similarity threshold; Step 43, according to and the road network distance of the sampling point pair , denotes the number of positive samples, denotes the number of negative samples, denotes the road network distance of the street sampling point pair in the positive sample, denotes the road network distance of the street sampling point pair in the negative sample; Step 44, constructing a graph structure , representing a node set composed of street sampling points, representing an edge set composed of street sampling point topological structures, representing a node attribute set composed of street sampling point feature vectors fused with multi-level physical semantics and spatial semantics; based on training a city street scene representation model using a graph convolutional neural network, and outputting a city street scene representation vector.

6. The method of claim 5, wherein, The formula of the similarity is: ， ； wherein, denotes a sample, denotes a number of samples, denotes Jenson’s Shannon divergence, denotes a multi-level functional semantic feature vector combination for a pair of sampling points.

7. The method of claim 5, wherein, The formula of the city street scene representation vector is: ； wherein, denotes a graph neural network model, denotes a first street sampling point through the fusion of multi-level physical semantics and spatial semantics to form a feature vector, denotes the number of street sampling points.

8. The method of claim 2, wherein, is to utilize image semantic segmentation algorithm to identify the visual element type in the street view image at each level of physical semantic and to count the pixel proportion of the visual element.

9. A system for representing urban street scenes based on multi-level semantic label augmentation, characterized in that, It comprises: a memory for storing instructions executable by the processor; a processor for executing the instructions to implement the city street scene representation method based on multi-level semantic label enhancement according to any one of claims 1-8.

10. A computer readable medium having stored thereon a computer program code, characterized in that, The computer program code implements the city street scene representation method based on multi-level semantic label enhancement according to any one of claims 1-8 when executed by the processor.

Images