Method and device for measuring element characteristics for shaping streetscapes

Abstract

The present invention provides a method and device for measuring element features for shaping streetscape appearance, which can be applied to the fields of image processing and urban and rural planning and design. The method comprises: processing streetscape images using a trained semantic segmentation model to obtain element type proportions and segmented images, wherein the element type proportions represent the area proportions of elements with element types in the streetscape images; processing streetscape images using a large language model to obtain streetscape description text; extracting features from the streetscape description text to obtain streetscape semantic features; fusing streetscape visual features and streetscape semantic features obtained by extracting feature types from the element type proportions and segmented images to obtain multimodal fusion features; inputting the multimodal fusion features into a trained evaluation model and outputting evaluation results, which include multiple evaluation values ​​for measuring the features of each element in the streetscape image according to different evaluation dimensions, and the evaluation results are used to shape streetscape appearance with local characteristics.

Description

Technical Field

[0001] The present invention relates to the fields of image processing and urban and rural planning and design, and in particular to a method and device for measuring element features for shaping block appearance. Background Art

[0002] During the period of rapid urbanization, urban landscapes are becoming increasingly homogeneous and lacking distinctive characteristics. Long-established regional and historical characteristics are facing unprecedented challenges. Measuring neighborhood landscape characteristics through technical means is an effective way to avoid a "sameness" among all cities and further explore highly recognizable cultural symbols across regions. It also supports understanding a city's historical heritage, shaping its visual identity, and showcasing and disseminating local and national culture in spatial design practices. However, existing methods for measuring neighborhood landscape rely on costly manual surveys, resulting in low efficiency, long processing times, and significant subjective bias, making them inadequate for landscape planning and management. Furthermore, mainstream model algorithms are limited to measuring neighborhood landscape through street view imagery recognition. However, these models are overly simplistic, lacking a deep integration of visual and semantic features within the images, and failing to simulate the complex cognitive processes of residents regarding the diverse characteristics of neighborhoods through visual perception. Consequently, they struggle to align with human perception of the environment, resulting in inaccurate and weakly interpretable measurement results that are insufficient for intelligently identifying and evaluating complex neighborhood landscapes. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method, device, electronic device, and computer program product for measuring element characteristics for shaping the streetscape.

[0004] According to a first aspect of the present invention, a method for measuring element features for shaping streetscape appearance is provided, comprising: processing streetscape images using a trained semantic segmentation model to obtain element type proportions and segmentation images, wherein the segmentation image represents a visual image of the segmentation result mapped to the streetscape image, and the element type proportion represents the area proportion of elements having the element type in the streetscape image; processing streetscape images using a large language model to obtain streetscape description text, wherein the streetscape description text includes an element attribute information description and an element spatial relationship description; performing feature extraction on the streetscape description text to obtain streetscape semantic features; fusing streetscape visual features and streetscape semantic features obtained by extracting feature types from the segmentation image to obtain multimodal fusion features; inputting the multimodal fusion features into a trained evaluation model to output an evaluation result, wherein the evaluation result includes multiple evaluation values ​​for measuring the features of each element in the streetscape image according to different evaluation dimensions, and the evaluation result is used to shape a streetscape appearance with local characteristics.

[0005] According to an embodiment of the present invention, a large language model is used to process a street view image to obtain street view description text, including: processing the street view image based on a target detection method to obtain multiple feature triplets, where the feature triplets represent feature type, bounding box coordinates, and confidence, and the street view image includes features of each of the multiple feature types; processing the street view image based on the feature triplets to determine a feature tree; and inputting an input sequence obtained based on the feature tree into the large language model to output street view description text.

[0006] According to an embodiment of the present invention, a street view image is processed according to feature triples to determine a feature tree, including: performing image capture on the street view image according to the feature triples to obtain a feature capture result; processing the feature capture result based on a target detection model to obtain feature attribute information features, the feature attribute information features including morphological features, color features, and material features of the target feature, and the feature capture result includes the target feature; and determining a feature tree based on the feature attribute information features and bounding box coordinates.

[0007] According to an embodiment of the present invention, a feature tree is determined based on feature attribute information features and bounding box coordinates, including: using a street view image as the root node of the feature tree, using the feature type in the street view image as the first-level child node, and using the feature attribute information features, bounding box coordinates, and feature type identifier corresponding to the feature type as the second-level child node; and determining the feature tree based on the root node, the first-level child node, and the second-level child node.

[0008] According to an embodiment of the present invention, the input sequence is obtained in the following manner: determining the bounding box coordinates corresponding to each of multiple feature types in the feature tree; determining the feature spatial relationship characteristics based on the multiple bounding box coordinates, the feature spatial relationship characteristics characterizing that there is a spatial relationship between feature types, and the spatial relationship includes covering, combination and mosaicking; determining the input sequence based on the feature type, feature attribute information characteristics and feature spatial relationship characteristics.

[0009] According to an embodiment of the present invention, feature extraction is performed on a street view description text to obtain street view semantic features, including: performing semantic preprocessing on the street view description text to obtain a target street view description text; performing word segmentation processing on the target street view description text to obtain a vocabulary list; and performing feature extraction on the vocabulary in the vocabulary list to obtain street view semantic features.

[0010] According to an embodiment of the present invention, the trained evaluation model is trained based on the following steps: obtaining a sample block style evaluation result corresponding to the sample street view image, the sample block style evaluation result includes a sample evaluation value of each evaluation dimension, and the sample evaluation value is used to measure the characteristics of each sample element in the sample street view image for the evaluation dimension; the sample street view image is binary classified according to the average value and variance of the multiple sample evaluation values ​​to obtain the classification value of the sample street view image; the binary classification label vector of the sample street view image is determined according to the classification value; the initial evaluation model is trained according to the sample street view image and the binary classification label vector to obtain the trained evaluation model.

[0011] According to an embodiment of the present invention, a trained semantic segmentation model is used to process a street view image to obtain a feature type ratio and a segmented image, including: using the trained semantic segmentation model to process the street view image to obtain a target segmentation result, where the target segmentation result represents a segmentation result of features having feature types in the street view image; calculating the feature type ratio based on the target segmentation result; and mapping the target segmentation result to the street view image to obtain a segmented image.

[0012] According to an embodiment of the present invention, a street view image includes a block street image and a building image; the street view image is obtained by: obtaining an initial street view image; and preprocessing the initial street view image to obtain the street view image, where the preprocessing includes at least one of adjusting brightness, adjusting contrast, removing noise, and normalizing.

[0013] The second aspect of the present invention provides an element feature measurement device for shaping streetscape appearance, including: an element type proportion module, which is used to use a trained semantic segmentation model to process streetscape images to obtain element type proportions and segmented images, where the segmented image represents a visual image of the segmentation result mapped to the streetscape image, and the element type proportion represents the area proportion of elements with the element type in the streetscape image; a streetscape description text acquisition module, which is used to use a large language model to process streetscape images to obtain streetscape description text, where the streetscape description text includes element attribute information description and element spatial relationship description; a streetscape semantic feature determination module, which is used to extract features from the streetscape description text to obtain streetscape semantic features; a fusion module, which is used to fuse streetscape visual features and streetscape semantic features obtained by extracting features from the element type proportions and the segmented image to obtain multimodal fusion features; an output module, which is used to input the multimodal fusion features into a trained evaluation model and output an evaluation result, where the evaluation result includes multiple evaluation values ​​for measuring the features of each element in the streetscape image according to different evaluation dimensions, and the evaluation result is used to shape a streetscape appearance with local characteristics.

[0014] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] The fourth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0016] According to an embodiment of the present invention, a trained semantic segmentation model is used to process street view images to obtain the element type ratio and segmented images; a large language model is used to process street view images to obtain street view description text, and feature extraction is performed on the street view description text to obtain street view semantic features; street view visual features and street view semantic features obtained by feature extraction of element type ratio and segmented images are fused to obtain multimodal fusion features; the multimodal fusion features are input into a trained evaluation model to output evaluation results, which include multiple evaluation values ​​for measuring the features of each element in the street view image according to different evaluation dimensions. The evaluation model can be used to identify and evaluate the urban block style. By establishing an effective fusion of multimodal features of block style, a fine-grained style measurement technology method suitable for interpreting the historical and cultural connotations of blocks is obtained, and an automated measurement technology system that is highly aligned with human subjective perception is established, significantly improving the dimensional integrity of block style measurement and the ability to analyze street view connotations. It can not only avoid the "sameness of thousands of cities", but also provide quantifiable and explainable intelligent decision-making support, further explore the highly recognizable cultural symbols of various places, and solve the core pain points of low efficiency, semantic fragmentation and poor interpretability of traditional methods, so as to build urban and rural landscapes with local characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A diagram showing an application scenario of the method and device for measuring element features for shaping a block appearance according to an embodiment of the present invention is shown;

[0018] Figure 2 A flow chart of a method for measuring element features for shaping a block appearance according to an embodiment of the present invention is shown;

[0019] Figure 3 A schematic diagram of determining a feature tree according to an embodiment of the present invention is shown;

[0020] Figure 4 shows a flow chart for determining an input sequence according to an embodiment of the present invention;

[0021] Figure 5 A flowchart of a method for determining street scene semantic features according to an embodiment of the present invention is shown;

[0022] Figure 6 A structural block diagram of an element feature measurement device for streetscape shaping according to an embodiment of the present invention is shown;

[0023] Figure 7 A block diagram of an electronic device suitable for implementing a method for measuring element features for shaping a streetscape is shown according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] During the rapid urbanization phase, urban landscapes are becoming increasingly similar and lacking distinctive features, leading to numerous problems such as a break in historical context. Identifying and evaluating urban landscapes is crucial for avoiding a "sameness" among cities and further exploring highly recognizable cultural symbols and spatial design vocabularies. Among these, neighborhood landscapes are a concentrated expression of historical heritage, urban memory, and regional culture. Identifying and evaluating neighborhood landscapes is an important way to explore the regularities of historical neighborhood streets and landscape organization, further understand the historical accumulation of urban development, and shape the city's visual characteristics.

[0029] In recent years, the development of urban streetscape data and computer image processing technology has provided new perspectives and methods for the identification and evaluation of historical neighborhood features. By processing and analyzing streetscape data through computer image processing, we can perform operations such as image recognition, classification, enhancement, and segmentation, accurately extracting information about the features of historical neighborhoods. This provides an efficient, intelligent, and precise approach to the preservation and display of local culture from a humanistic perspective.

[0030] Currently, the commonly used methods for identifying and evaluating the appearance of historical blocks include survey-based methods, indirect simulation perception-based methods, and deep learning-based methods. Survey-based methods rely on manual field surveys, measurements, and expert evaluations, which are inefficient, costly, and highly subjective, making them difficult to process large-scale historical block data. Indirect simulation perception-based methods construct appearance scenes through modeling and simulate real-world environments with virtual devices, which can effectively shorten the survey cycle. However, they are limited in terms of interaction methods, restoration degree, and field of view, and the interpretability of identification and evaluation results is insufficient. Deep learning-based methods can quickly obtain large-scale block appearance information and can simulate human vision to a high degree by adjusting the image acquisition angle. However, existing research still has three major problems: First, existing research is mostly generalized to urban-scale streetscape appearance evaluation, lacking an indicator system and streetscape dataset for appearance identification and evaluation that interprets the historical and cultural connotations of blocks, making it difficult to meet the current requirements for the protection, renewal, and management of block historical and cultural landscapes under the current refined development. Second, existing technologies focus more on the direct measurement of the external characteristics of physical elements, less on the physiological visual emphasis and psychological perception logic of human cognition of appearance, and lack human psychological understanding. Perception is coupled to the material space environment; thirdly, the existing technical methods applied to street view image processing are relatively backward. The mainstream model algorithm lacks deep integration between visual and semantic information, and only performs surface segmentation of image visual elements. When faced with street view image content with similar ground objects, it lacks recognition of the color, material and other attributes of different elements, making it difficult to accurately depict the correlation between different elements such as covering, combination, and mosaic. The synergy of multimodal information has not been fully explored, and there is a lack of accurate interpretation of the cultural connotation and social information of the block style in street view images, which cannot well meet the needs of complex intelligent style recognition and evaluation tasks.

[0031] In view of this, the present invention provides a method and device for measuring element features for shaping streetscape appearance. The method includes: using a trained semantic segmentation model to process streetscape images to obtain element type ratios and segmentation images, the segmentation image represents a visual image of the segmentation result mapped to the streetscape image, and the element type ratio represents the area ratio of elements with the element type in the streetscape image; using a large language model to process streetscape images to obtain streetscape description text, the streetscape description text includes element attribute information description and element spatial relationship description; feature extraction of the streetscape description text to obtain streetscape semantic features; fusing streetscape visual features and streetscape semantic features obtained by feature extraction of element type ratios and segmentation images to obtain multimodal fusion features; inputting the multimodal fusion features into a trained evaluation model and outputting evaluation results, the evaluation results including multiple evaluation values ​​for measuring the features of each element in the streetscape image according to different evaluation dimensions, and the evaluation results are used to shape streetscape appearance with local characteristics.

[0032] In the technical solution of the present invention, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] In scenarios where personal information is used for automated decision-making, the methods, devices, electronic devices, and program products provided by the embodiments of the present invention all provide users with corresponding operation portals for them to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge, and skills, and have reached a certain level of professionalism.

[0034] Figure 1 A diagram showing an application scenario of a method and apparatus for measuring element features for shaping a block appearance according to an embodiment of the present invention is shown.

[0035] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0036] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0037] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0038] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0039] It should be noted that the element feature measurement method for block appearance shaping provided by the embodiment of the present invention can generally be executed by the server 105. Accordingly, the element feature measurement device for block appearance shaping provided by the embodiment of the present invention can generally be set in the server 105. The element feature measurement method for block appearance shaping provided by the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the element feature measurement device for block appearance shaping provided by the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0040] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0041] Figure 2 A flowchart of a method for measuring element characteristics for shaping a block appearance according to an embodiment of the present invention is shown.

[0042] like Figure 2 As shown, the element feature measurement method for streetscape shaping of this embodiment includes operations S210 to S250, and the element feature measurement method for streetscape shaping of this embodiment can be executed by an electronic device.

[0043] In operation S210 , the trained semantic segmentation model is used to process the street view image to obtain feature type ratios and a segmented image.

[0044] In operation S220 , the street view image is processed using the large language model to obtain street view description text.

[0045] In operation S230 , feature extraction is performed on the street view description text to obtain street view semantic features.

[0046] In operation S240 , the street view visual features and the street view semantic features obtained by extracting the feature proportions of the element types and the segmented image are fused to obtain a multimodal fusion feature.

[0047] In operation S250 , the multimodal fusion features are input into the trained evaluation model, and an evaluation result is output.

[0048] According to an embodiment of the present invention, a semantic segmentation model may include a fully convolutional network, a pyramid scene parsing network, etc., but is not limited thereto. The embodiments of the present invention do not limit the semantic segmentation model. Street view images may include images covering the main streets and important buildings of a block to comprehensively reflect the appearance of the block. The segmented image represents a visual image of the segmentation result mapped to the street view image. Feature types may include various building components and structures such as columns, arches, and roofs in the block appearance scene. The feature type ratio represents the area ratio of features with feature types in the street view image. In image processing software, different features are marked with different colors, and the semantic segmentation results are rendered as a red-green-blue (RGB) image to obtain a visual image (i.e., a segmented image) of the segmentation result mapped to the original image.

[0049] For example, a street view image includes a first column having a column type, a first arch having a second arch having an arch type, and a first roof having a second roof having a roof type. The feature type ratio based on the column type can represent the ratio of the area of ​​the first column in the street view image to the area of ​​the street view image, the feature type ratio based on the arch type can represent the ratio of the sum of the areas of the first arch and the second arch in the street view image to the area of ​​the street view image, and the feature type ratio based on the roof type can represent the ratio of the sum of the areas of the first roof and the second roof in the street view image to the area of ​​the street view image.

[0050] According to an embodiment of the present invention, a street view description may include feature attribute information and feature spatial relationship descriptions. Feature attribute information may include the color, shape, and material of the feature type. Feature spatial relationship descriptions may include associations between the feature type and other feature types, such as overlapping, combination, and mosaicking.

[0051] According to embodiments of the present invention, street view semantic features can be determined based on street view description text. A text embedding model can be used to perform word embedding processing on the street view description text, converting the street view description text into vectors in a high-dimensional space. These vectors can capture the semantic information of the street view description text. The standardized feature type ratio data can be associated with the text vectors according to identifiers, thereby forming street view semantic features that contain the semantic information of the text.

[0052] According to an embodiment of the present invention, principal component analysis can be used to process element type proportions and convert them into element type proportion features to extract primary features and remove redundant information. The element type proportions can be processed based on principal component analysis to determine relatively important element types and less important element types within the element types. The element type proportions corresponding to the relatively important element types can be processed as primary features, while the element type proportions corresponding to the less important element types can be removed as redundant information.

[0053] According to an embodiment of the present invention, feature extraction can be performed on the element type ratio and the segmentation result mapped to the visual image of the original image to obtain street view visual features. When extracting the vector of the element type ratio, the element type ratio data can be normalized to obtain standardized element type ratio data. For each street view image, the corresponding ratio values ​​are sequentially arranged and combined into a vector according to the determined element category order to obtain feature vectors of different element ratios. Normalization is performed for subsequent feature fusion. When extracting the vector of the segmentation result mapped to the visual image of the original image, an image encoder can be selected for conversion processing. The conversion process includes format conversion, resizing, normalization, feature extraction, and vector conversion operations. For example, the street view image can be converted to red, green, and blue (RGB) format and resized to 224×224 pixels for input. Normalize the image pixel values, setting the mean to [0.485, 0.456, 0.406] and the standard deviation to [0.229, 0.224, 0.225] to improve model training and inference stability. Load pretrained weights for transfer learning, setting the hidden layer dimension to 768, the number of heads in the multi-head attention mechanism to 12, and the number of stacked Transformer modules to 12. The preprocessed street scene semantic segmentation images are fed into the image encoder batch by batch, with a batch size of 2, for feature extraction and vector conversion. When fusion of two vectors, the query vector is a vector mapping the segmentation result to the original image visualization, and the key-value pair is a vector of feature type proportions. Multi-head attention is used to calculate the association weights between the two vectors, and a concatenation function is used to fuse them into a joint vector based on the association weights to obtain the street scene visual features. The street view visual features are obtained by joint vectors, avoiding the loss of visual structure information of street view images or the proportion of different elements, and realizing comprehensive representation of street view visual features.

[0054] According to embodiments of the present invention, street view visual features and street view semantic features can be fused to generate multimodal fusion features. A multi-head attention mechanism can be used as a cross-modal attention layer to calculate the attention weight of each head and merge the outputs of all attention heads to generate multimodal fusion features.

[0055] According to an embodiment of the present invention, multimodal fusion features can be input into a trained evaluation model to output an evaluation result. The evaluation result can include multiple evaluation values ​​for measuring the characteristics of each element in the street view image according to different evaluation dimensions. For example, the evaluation result can include an evaluation value for the landscape uniqueness of each element in the street view image. If the evaluation value is larger, it can be said that the element in the street view image can better shape the streetscape with local characteristics. Measuring features in different evaluation dimensions can reduce subjective scoring fluctuations and effectively improve the accuracy of landscape recognition and the comprehensiveness of evaluation.

[0056] According to an embodiment of the present invention, a trained semantic segmentation model is used to process street view images to obtain the element type ratio and segmented images; a large language model is used to process street view images to obtain street view description text, and feature extraction is performed on the street view description text to obtain street view semantic features; street view visual features and street view semantic features obtained by feature extraction of element type ratio and segmented images are fused to obtain multimodal fusion features; the multimodal fusion features are input into a trained evaluation model to output evaluation results, which include multiple evaluation values ​​for measuring the features of each element in the street view image according to different evaluation dimensions. The evaluation model can be used to identify and evaluate the urban block style. By establishing an effective fusion of multimodal features of block style, a fine-grained style measurement technology method suitable for interpreting the historical and cultural connotations of blocks is obtained, and an automated measurement technology system that is highly aligned with human subjective perception is established, significantly improving the dimensional integrity of block style measurement and the ability to analyze street view connotations. It can not only avoid the "sameness of thousands of cities", but also provide quantifiable and explainable intelligent decision-making support, further explore the highly recognizable cultural symbols of various places, and solve the core pain points of low efficiency, semantic fragmentation and poor interpretability of traditional methods, so as to build urban and rural landscapes with local characteristics.

[0057] According to an embodiment of the present invention, a large language model is used to process street view images to obtain street view description text, including: processing the street view image based on a target detection method to obtain multiple feature triplets, wherein the feature triplets represent feature type, bounding box coordinates, and confidence, and the street view image includes elements of each of the multiple feature types; processing the street view image according to the feature triplets to determine a feature tree; and inputting the input sequence obtained based on the feature tree into the large language model to output the street view description text. The feature tree can highlight important historical and cultural elements in the street view image and their hierarchical relationships, explicitly encode the geometric distribution and semantic associations of visual elements, and improve the disconnection and lack of systematicity between visual features and semantic descriptions in traditional methods. At the same time, it provides the large language model with an input sequence with spatial topological constraints, so that the feature attributes and spatial relationship descriptions it generates are more consistent with the contextual logic of the real scene, and improve the description distortion caused by semantic drift problems in traditional end-to-end image-text models.

[0058] According to an embodiment of the present invention, a feature triple represents a feature type, a bounding box coordinate, and a confidence level. Buildings in street view images may include multiple feature types, such as arches, roofs, windows, etc., but are not limited thereto. The embodiment of the present invention does not list all feature types. By processing street view images based on the target detection method, multiple feature triplets can be obtained. Let the feature triple set be 𝒯, where c i is the category of the i-th element, b i is the bounding box coordinate of the i-th feature, s i is the detection confidence of the i-th element, as shown in formula (1). The bounding box coordinates include four parts, as shown in formula (2).

[0059] (1);

[0060] =(x min ,y min ,x max ,y max )(2);

[0061] Among them, x min is the minimum value of the horizontal coordinate of the image, y min is the minimum value of the image vertical coordinate, x max is the maximum value of the horizontal coordinate of the image, y max is the maximum value of the image's vertical coordinate.

[0062] Specifically, a streetscape dataset with feature type identification labels can be used to perform feature type detection and region extraction using an object detection model. The Efficient Multi-Scale Attention Module (EMA) can also be embedded, with a channel attention weight threshold set to 0.35 to enhance the detection of small features such as capitals. A dual-path feature pyramid can also be constructed to improve the recognition accuracy of dense feature types such as continuous arcades. The output consists of a triplet of <feature type, bounding box coordinates, and confidence level>.

[0063] According to an embodiment of the present invention, a street view image can be processed based on feature triples to determine a feature tree. An input sequence obtained based on the feature tree is input into a large language model to output a street view description text.

[0064] According to embodiments of the present invention, the relative positions of elements of a feature type can be calculated using the coordinates of the target detection frame, describing relationships such as covering, combining, and inlaying. Covering relationships can indicate that the position of the first element covers the position of the second element, combining relationships can indicate that the third and fourth elements are connected using a concave-convex combination, and inlaying relationships can indicate that the position of the fifth element is inlaid within the position of the sixth element. A caption describing a street scene, for example, might read "The arcade istopped with a continuous sloping roof." A weight mapping can be set to weight importance, increasing the generation weight for landmark components (such as arches) and simplifying the description of minor components (pavement). For example, if the same weight is applied to landmark and minor components, when generating the street scene description text, the description ratio of the minor components will be the same as that of the landmark components, which is detrimental to the subsequent generation of evaluation results. The weight mapping can be in the form of a table, a set, or other form, and is not limited here.

[0065] According to an embodiment of the present invention, 𝓡 is defined as the spatial relationship between elements, including inlay, occlusion, combination, or non-existence (otherwise). In the calculation, they are encoded as 1, 2, 3, and 0 respectively, as shown in formula (3). Suppose the set of all detected elements is E={e1,e2,......,e n}, the i-th element e i By its bounding box B i and category c i Definition, where is the bounding box B i The bounding box coordinates of the jth feature e j By its bounding box B j and category c j Definition, where is the bounding box B j When a pair of features satisfies multiple relationships at the same time, they are sorted by priority based on Inlay > Occlusion > Combination.

[0066] (3);

[0067] If element e i Completely located in e j Internal, and the category belongs to the subordinate relationship, it is called the mosaic relationship, the category subordinate relationship is defined by the knowledge base (such as windows belong to walls), where Indicates that the bounding box completely contains IsChild is the category membership judgment function, as shown in formula (4).

[0068] (4);

[0069] If element e j The bounding box of i The bounding box of is partially covered, and the coverage area exceeds the coverage threshold, which is called an occlusion relationship. occ is the masking threshold, as shown in formula (5).

[0070] (5);

[0071] If two elements are adjacent in space and meet specific geometric connection conditions (such as common edges and collinearity), the connectivity is measured. If the connectivity exceeds the connectivity threshold, it is called a combination relationship, where is the edge of the feature's bounding box, is the feature's bounding box perimeter, conn is the connectivity threshold, as shown in formulas (6) and (7).

[0072] (6);

[0073] (7);

[0074] According to an embodiment of the present invention, a holistic description generation strategy can be applied, invoking a model to generate street scene description text. Inference code can be written using the model's inference application programming interface (API), and the constructed input sequence can be passed to a large language model. The model can then be fine-tuned to accommodate architectural terminology, establishing an architectural knowledge constraint library to eliminate semantic ambiguity and avoid nonsensical descriptions such as "wooden arches." You can set generation parameters such as the length limit of the generated text and temperature parameters, design prompt words, and formulate description rules. The basic description layer uses feature-template mapping rules to generate single-component descriptions. The relationship description layer parses spatial encoding to generate orientation descriptions. The overall induction layer counts the feature distribution of similar components. Finally, a scene-level summary is generated to obtain street scene description text, such as "The image contains a light gray (RGB value 192, 192, 192) stone semi-circular arch, with dark brown brick columns on both sides (RGB values ​​100, 51, 53). Typical local binary pattern (LBP) texture features (entropy value 7.82) are visible on the arch surface, and there is a color difference of ΔE=4.2 with the adjacent columns. The arch, semi-circular in shape, is constructed with bricks in Deep Walnut Brown, Pantone 19-1012."

[0075] According to an embodiment of the present invention, the quality of the generated street view description text can also be evaluated and optimized. 20% of the street view description text can be randomly sampled, and the style accuracy, element integrity, and spatial consistency can be manually evaluated, and then targeted optimization adjustments can be made. Style accuracy can refer to whether the generated street view description text conforms to the architectural style, element integrity can refer to whether the generated street view description text covers all important elements of the street view, and spatial consistency can refer to whether the generated street view description text is logically coherent in space, that is, whether the order and direction of the description are reasonable and whether they conform to the actual spatial layout.

[0076] According to the embodiments of the present invention, traditional methods are limited to the simple segmentation of physical environmental elements and lack the coupling supplement of attribute information such as vision and space. The present invention can convert discrete element features into a structured knowledge system that can support semantic analysis, and upgrade the pixel-level output of the traditional segmentation model into graph structure data that can support semantic reasoning, thereby realizing the unified expression of multimodal features and the generation of interpretable semantics, thereby solving the standardization problem of heterogeneous data such as morphology, color, and material, providing a multimodal data base that is machine-computable and human-understandable, and improving the semantic gap problem in traditional style measurement.

[0077] According to an embodiment of the present invention, street view images are processed based on feature triples to determine a feature tree, including: performing image capture on the street view image based on the feature triples to obtain feature capture results, thereby improving the stability of feature detection under complex viewing angles; processing the feature capture results based on a target detection model to obtain feature attribute information features, wherein the feature attribute information features include the morphological features, color features, and material features of the target feature, and the feature capture results include the target feature; and determining a feature tree based on the feature attribute information features and bounding box coordinates. This is conducive to in-depth analysis of the local semantics of local historical culture in street view features. It retains the effective morphological, color, and material differences discernible to the human eye, and improves the defect of losing important detail information in traditional global feature extraction.

[0078] According to embodiments of the present invention, feature type image capture and preprocessing can be performed based on feature triples, followed by adaptive image segmentation to obtain feature segmentation results, which contain target features. Line detection algorithms and geometric constraints can be used to automatically calibrate feature types in the original street view image. For example, if the target feature is a roof, the four vertices of the roof area and the intersection of the wall baseline and the ground can be calibrated. Coordinate correction is triggered when the viewing angle is greater than 45°, and image correction is performed based on the coordinate system defined by the inverse perspective transformation, reducing recognition errors caused by different viewing angles of the street view image. is the viewing angle (such as the angle between the shooting angle and the building facade normal vector), as shown in formula (8), is the building facade normal vector, is the viewing direction vector of the street view image, when >45° triggers inverse perspective transformation.

[0079] (8);

[0080] is the corrected local image of the element, as shown in formula (9), i is the i-th element, is the inverse perspective transformation function, is the original street view image, are the bounding box coordinates, is the viewing angle.

[0081] (9);

[0082] According to an embodiment of the present invention, feature attribute information features can be obtained by processing feature interception results based on a target detection model. Feature attribute information features may include morphological features, color features, and material features of the target feature. Feature type features and semantic concepts can be encoded, and morphological, color, and material features can be annotated. Arches are classified and labeled as semicircular arches, pointed arches, and horseshoe arches; roofs are classified and labeled as flat roofs, sloping roofs, domes, and spire; windows are classified and labeled as square windows and round windows; and walls are classified and labeled as solid walls and hollow walls, generating morphological features containing the confidence of the main category. is the morphological feature vector, as shown in formula (10), is the ResNet feature extraction function, is the contour polygon fitting function, is the feature contour point set, It is a feature splicing operation.

[0083] (10);

[0084] The RGB color space is converted into a color space that conforms to the human eye's perception characteristics (CIE-Lab). A clustering algorithm is used to obtain component image color space clustering, retaining color clusters and their proportions with ΔE less than 5 to obtain color features. The main color tones of each component are solved through clustering optimization, retaining effective color differences discernible by the human eye, and solving the problem of color information loss in global feature extraction. is the color feature vector, as shown in formula (11), is the cluster center of the CIE-Lab color space, represents three-dimensional space, represents the Euclidean distance, is a pixel in the street view image, is the value of the pixel in CIE-Lab space.

[0085] (11);

[0086] Texture features are extracted to generate a 59-dimensional texture histogram to quantify the physical texture properties of the component surface in the image. The mean texture energy of the Gray-Level Co-occurrence Matrix (GLCM) in four directions is calculated to enhance the rotational invariance of material recognition. is the texture feature vector, as shown in formula (12), is the gray-level co-occurrence matrix, is the texture analysis direction, It is the texture energy calculation function.

[0087] (12);

[0088] The 2048-dimensional deep features are extracted through the pre-trained residual network, as shown in formula (13).

[0089] (13);

[0090] in, Represents the depth feature vector of the i-th component, with a dimension of 2048, represents the pre-trained residual network, Represents the rectified local image of the feature.

[0091] Construct a feature splicing vector and input it into the fine-tuned classifier to classify the wall into stone, brick, wood, and paint; the fence into stone, brick, wood, and metal; and the pavement into stone, brick, and concrete to obtain material features.

[0092] (14);

[0093] in, represents the fused feature vector, represents the texture feature vector, represents the depth feature vector of the i-th component, It represents the fusion feature of 59-dimensional texture features and 2048-dimensional depth features.

[0094] (15);

[0095] in, represents the material feature vector, represents the normalized exponential function, which is used to map the linear score to a [0,1] probability distribution. represents the weight matrix, represents the bias vector, Indicates the material category.

[0096] According to embodiments of the present invention, a feature tree can be determined based on feature attribute information and bounding box coordinates. A feature tree can be constructed for structured feature representation, employing a hierarchical nested data structure of attribute-value pairs. The root node can record global identification information corresponding to the street view image. First-level child nodes establish branches based on detected features, such as arches, columns, and roofs. Second-level child nodes record feature sets such as unique feature type identifiers, geometric attributes such as bounding box coordinates, and feature attribute information feature tags.

[0097] According to the embodiments of the present invention, when faced with street view image content with similar ground object components, the present invention proposes the recognition of attributes such as color and material of different elements, which can more accurately depict the association relationships such as covering, combination, and mosaic between different elements, and help to improve the limitations of traditional features in plane representation.

[0098] According to an embodiment of the present invention, a feature tree is determined based on feature attribute information and bounding box coordinates. This includes: using the street view image as the root node of the feature tree, the feature type in the street view image as the first-level child node, and the feature attribute information, bounding box coordinates, and feature type identifier corresponding to the feature type as the second-level child node; and determining the feature tree based on the root node, first-level child nodes, and second-level child nodes. By constructing the feature tree, a coherent and structured storage of feature attributes and spatial relationships is achieved, supporting multi-granularity queries on massive feature information in street view images and optimizing the information loss caused by the existing single-planar graph structure. is the feature tree graph structure, as shown in formula (16). is the set of all nodes, as shown in formula (17). is the connecting edge between nodes, as shown in formula (25). For images, is the total number of parts detected.

[0099] (16);

[0100] is the root node, as shown in formula (18), the image The hash value (Hash) is generated. is the first-level node of the i-th component, describing the feature type, as shown in formula (19). is the second-level child node of the i-th component, describing the feature attribute information characteristics, bounding box coordinates and feature type identification, as shown in formula (20).

[0101] (17);

[0102] (18);

[0103] The first level nodes include categories , Bounding Box , confidence

[0104] (19);

[0105] In the second-level child node, k is the k-th attribute of the i-th component. is the geometric attribute node of the i-th component, as shown in formula (21), including the bounding box and unique identifier , is the morphological attribute node of the i-th component, as shown in formula (22), is the color attribute node of the i-th component, as shown in formula (23), is the material attribute node of the i-th component, as shown in formula (24).

[0106] (20);

[0107] (twenty one);

[0108] (twenty two);

[0109] (twenty three);

[0110] (twenty four);

[0111] In the connecting edges between nodes, is a hierarchical edge that describes the connection between the subordinate relationships between nodes, as shown in formula (26), is a topological edge that describes the connection of the spatial topological relationship between nodes, as shown in formula (27).

[0112] (25);

[0113] Level edge, is the root node, is the first-level component layer node, is the set of all first-level child nodes, is the second-level attribute layer node, The collection of all second-level child nodes.

[0114] (26); in topological edges, 、 For different nodes at the same level, is the spatial relationship between two nodes.

[0115] (27);

[0116] According to an embodiment of the present invention, a standard component type dictionary can be defined according to a two-level coding strategy at the component level and attribute level (for example, element type = ["semicircular arch", "pointed arch", "horseshoe arch"]), a component unique identification code can be generated, for example, 001, features can be converted into standardized parameters, and annotation information can be combined to serve as first-level child nodes and second-level child nodes.

[0117] Figure 3 A schematic diagram of determining a feature tree according to an embodiment of the present invention is shown.

[0118] like Figure 3 As shown, the pre-processed street view image 301 can be used as the root node, and the feature types of the semicircular arch 302, the pointed arch 303, and the horseshoe arch 304 are included in the street view image 301. The feature types of the semicircular arch 302, the pointed arch 303, and the horseshoe arch 304 are used as first-level child nodes. Each first-level child node includes corresponding feature attribute information features, bounding box coordinates, and feature type identifiers. The corresponding feature attribute information features, bounding box coordinates, and feature type identifiers are used as second-level child nodes. The semicircular arch 302 includes corresponding feature attribute information features 305, bounding box coordinates 306, and feature type identifier 307. The pointed arch 303 includes corresponding feature attribute information features 308, bounding box coordinates 309, and feature type identifier 310. The horseshoe arch 304 includes corresponding feature attribute information features 311, bounding box coordinates 312, and feature type identifier 313.

[0119] Figure 4 A flowchart of determining an input sequence according to an embodiment of the present invention is shown.

[0120] like Figure 4 As shown, the method for determining an input sequence in this embodiment includes operations S410 to S430, and the element feature measurement method for shaping a streetscape can be executed by an electronic device.

[0121] In operation S410 , bounding box coordinates corresponding to each of a plurality of feature types in a feature tree are determined.

[0122] In operation S420 , a feature of element spatial relationship is determined based on the plurality of bounding box coordinates. The feature of element spatial relationship indicates that there is a spatial relationship between element types. The spatial relationship includes covering, combining, and mosaicking.

[0123] In operation S430, an input sequence is determined based on the feature type, feature attribute information characteristics, and feature spatial relationship characteristics.

[0124] According to an embodiment of the present invention, the bounding box coordinates corresponding to each of multiple feature types in a feature tree can be determined. Based on the multiple bounding box coordinates, the spatial relationship characteristics of the features can be determined. The spatial relationship can include covering, combining, and mosaicking.

[0125] For example, the bounding box coordinates of the spire are [0, 0], [0, 10], [10, 0], [10, 10], and the bounding box coordinates of the roof are [0, 1], [-1, -3], [5, 1], [3, -3]. Therefore, the spatial relationship between the two is an overlapping relationship. For example, the coordinates of the tenon in a building are [0, 0], [0, 2], [1, 0], [1, 1], [2, 0], [2, 1], [3, 0], [3, 2], and the coordinates of the mortise are [0, 0], [1, 0], [1, 1], [2, 0], [2, 1], [3, 0], [3, -2], [0, -2]. Therefore, the spatial relationship between the two is a combination relationship. For example, the coordinates of a square window in a building are [2, 2], [2, 4], [4, 2], [4, 4], and the coordinates of a stone wall are [0, 0], [10, 0], [10, 10], [10, 0]. It can be seen that the spatial relationship between the two is a mosaic relationship. The element spatial relationship feature indicates that there is a spatial relationship between element types, but is not limited to this. The embodiments of the present invention are not limited to this.

[0126] According to embodiments of the present invention, an input sequence can be determined based on feature type, feature attribute information, and feature spatial relationship characteristics. This converts feature tree information into a sequential instruction stream understandable by a large language model (LLM), optimizing existing model text generation issues such as semantic drift and incomplete features. The input sequence can be defined as <feature type, target feature, target feature's feature color, target feature's feature morphology, target feature's feature material, and the spatial relationship characteristics between the target feature and other features>.

[0127] Figure 5 A flowchart of a method for determining street view semantic features according to an embodiment of the present invention is shown.

[0128] like Figure 5 As shown, the street view semantic feature determination method of this embodiment includes operations S510 to S530, and the element feature measurement method for street view style shaping can be executed by an electronic device.

[0129] In operation S510 , semantic preprocessing is performed on the street view description text to obtain a target street view description text.

[0130] According to embodiments of the present invention, semantic preprocessing can be performed on street scene description text to obtain target street scene description text. For example, stop words such as "the" and "an" and punctuation marks can be removed, and stemming or lemmatization can be performed, such as restoring "running" to "run," to reduce noise, improve the accuracy of semantic capture, and perform subsequent word segmentation processing.

[0131] In operation S520 , word segmentation is performed on the target street view description text to obtain a vocabulary list.

[0132] According to an embodiment of the present invention, the target street scene description text can be segmented using a custom dictionary to obtain a vocabulary list. The vocabulary list can include two columns: vocabulary and word frequency. A word frequency count can also be performed on the vocabulary list to calculate the frequency of each word in the text, providing a weight basis for subsequently obtaining multimodal fusion features.

[0133] In operation S530 , feature extraction is performed on the words in the vocabulary list to obtain street view semantic features.

[0134] According to an embodiment of the present invention, an embedding model can be used to perform word embedding processing on the words in the vocabulary list to obtain street scene semantic features. The text is converted into vectors in a high-dimensional space. These vectors can capture the semantic information of the text.

[0135] According to an embodiment of the present invention, the trained evaluation model is trained based on the following steps: obtaining a sample block style evaluation result corresponding to the sample street view image, the sample block style evaluation result includes a sample evaluation value of each evaluation dimension, and the sample evaluation value is used to measure the characteristics of each sample element in the sample street view image for the evaluation dimension; the sample street view image is binary classified according to the average value and variance of the multiple sample evaluation values ​​to obtain the classification value of the sample street view image; the binary classification label vector of the sample street view image is determined according to the classification value; the initial evaluation model is trained according to the sample street view image and the binary classification label vector to obtain the trained evaluation model.

[0136] According to embodiments of the present invention, a sample block appearance evaluation result corresponding to a sample street view image can be obtained. The sample block appearance evaluation result can include sample evaluation values ​​for multiple evaluation dimensions. For example, the multiple evaluation dimensions may include overall evaluation, historical and cultural significance, appearance coordination, element richness, landscape uniqueness, and pleasure and comfort. The overall evaluation can represent the overall impression and comprehensive experience of the street view image, or it can be the average of the sample evaluation values ​​of other evaluation dimensions for the same street view image. Historical and cultural significance can represent the historical and cultural value embodied by elements such as historical buildings, relics, and cultural symbols in the street view image. Appearance coordination can indicate whether the style, color, and scale of the buildings, streets, and greenery in the street view are coordinated and unified. Element richness can indicate the types and number of different elements in the street view image, thereby deriving a diversity index. Landscape uniqueness can be used to assess the uniqueness of the street view image compared to other street view images, including architectural style, layout, and cultural characteristics. Pleasure and comfort can be used to measure the visual and psychological pleasure and comfort brought to people by the street view corresponding to the street view image, but is not limited to this and is not limited in this regard by the embodiments of the present invention. The sample evaluation value can be used to measure the characteristics of each sample element in the sample street view image with respect to the evaluation dimension.

[0137] According to embodiments of the present invention, a sample street view image can be binary-classified based on the mean and variance of multiple sample evaluation values ​​to obtain a classification value for the sample street view image. A binary-class label vector for the sample street view image can be determined based on the classification value and a preset threshold. The binary-class label vector for each sample street view image can be obtained and used as a training set to train an initial evaluation model.

[0138] According to an embodiment of the present invention, an initial evaluation model is trained based on sample street view images and binary classification label vectors to obtain a trained evaluation model.

[0139] According to the embodiments of the present invention, the present invention changes the single rating dimension, supplements the subjective evaluation indicators for historical style, reduces the fluctuation of human subjective scores, shifts from "experience-driven" to "data-driven", optimizes the business flow of urban planning work in style evaluation, and realizes the conversion of machine features into style descriptions that can be understood by humans, effectively improving the accuracy of style recognition and the comprehensiveness of evaluation.

[0140] According to an embodiment of the present invention, a trained semantic segmentation model is used to process a street view image to obtain a feature type ratio and a segmented image. The method includes: processing the street view image using the trained semantic segmentation model to obtain a target segmentation result, wherein the target segmentation result represents a segmentation result of features having feature types in the street view image; calculating the feature type ratio based on the target segmentation result; and mapping the target segmentation result to the street view image to obtain a segmented image.

[0141] According to an embodiment of the present invention, in image processing software, different elements are marked with different colors, and the semantic segmentation result is rendered as an RGB image to obtain a visual image (segmented image) in which the segmentation result is mapped to the original image.

[0142] According to embodiments of the present invention, a trained semantic segmentation model can be used to process street view images to obtain target segmentation results. The target segmentation results represent the segmentation results of features with feature types in the street view image. Alternatively, 21 major building components and structures in street scene scenes, such as columns, arches, and roofs, can be selected as object classes and the Segment Anything Model 2 (SAM2) can be used to perform preliminary image segmentation to identify approximate regions and outlines. Annotation tools can be used for precise annotation and format conversion to construct a street view dataset with feature type identification labels. The annotated street view dataset can be allocated in an 8:1:1 ratio: training, validation, and test sets. Parameters such as the learning rate, number of iterations, and optimizer type are configured for model training. The training set data is then input into the semantic segmentation model for training, resulting in a model that effectively identifies feature types in street view images.

[0143] According to an embodiment of the present invention, a trained semantic segmentation model is used to perform semantic segmentation on street view images, identifying and segmenting 21 types of physical elements within the image, such as various building components and structures. Different objects in the image are distinguished by different colors. Based on the target segmentation results, the area percentage of each element type in the image is calculated, and a visualization image is generated by mapping the segmentation results to the original image for use in street view assessment.

[0144] According to an embodiment of the present invention, a street view image includes a block street image and a building image; the street view image is obtained by: obtaining an initial street view image; and preprocessing the initial street view image to obtain the street view image, where the preprocessing includes at least one of adjusting brightness, adjusting contrast, removing noise, and normalizing.

[0145] According to embodiments of the present invention, street view images can include images of neighborhood streets and buildings. Initial street view images can be obtained using the Street View map service. Road network data is obtained through Open Street Map, and street view points are generated using geographic information system software. The initial street view images are then retrieved using an API. When collecting data, ensure that the main streets and important buildings in the neighborhood are covered to fully reflect the neighborhood's appearance.

[0146] According to embodiments of the present invention, an initial street view image is preprocessed to obtain a street view image. This preprocessing may include at least one of brightness adjustment, contrast adjustment, noise removal, and normalization. A quality check is performed on the obtained initial street view image to remove blurry, improperly exposed, or incomplete images. Subsequently, image brightness and contrast are adjusted, noise is removed, and preprocessing operations such as image scaling, cropping, and normalization are performed to resize the initial street view image to the size and format required for network input.

[0147] Figure 6 A structural block diagram of an element feature measurement device for streetscape shaping according to an embodiment of the present invention is shown.

[0148] like Figure 6 As shown, the element feature measurement device for streetscape shaping in this embodiment includes an element type proportion module 610, a streetscape description text acquisition module 620, a streetscape semantic feature determination module 630, a fusion module 640 and an output module 650.

[0149] The feature type ratio module 610 is used to process the street view image using the trained semantic segmentation model to obtain the feature type ratio and the segmentation image. The segmentation image represents the segmentation result mapped to the visual image of the street view image, and the feature type ratio represents the area ratio of the features of the feature type in the street view image.

[0150] The street view description text obtaining module 620 is used to process the street view image using a large language model to obtain street view description text, which includes feature attribute information description and feature spatial relationship description.

[0151] The street view semantic feature determination module 630 is used to extract features from the street view description text to obtain street view semantic features.

[0152] The fusion module 640 is used to fuse the street view visual features and street view semantic features obtained by extracting the feature proportions of element types and the segmented images to obtain multimodal fusion features.

[0153] Output module 650 is used to input the multimodal fusion features into the trained evaluation model and output the evaluation results. The evaluation results include multiple evaluation values ​​for measuring the features of each element in the street view image according to different evaluation dimensions. The evaluation results are used to shape the streetscape with local characteristics.

[0154] According to an embodiment of the present invention, a trained semantic segmentation model is used to process street view images to obtain the element type ratio and segmented images; a large language model is used to process street view images to obtain street view description text, and feature extraction is performed on the street view description text to obtain street view semantic features; street view visual features and street view semantic features obtained by feature extraction of element type ratio and segmented images are fused to obtain multimodal fusion features; the multimodal fusion features are input into a trained evaluation model to output evaluation results, which include multiple evaluation values ​​for measuring the features of each element in the street view image according to different evaluation dimensions. The evaluation model can be used to identify and evaluate the urban block style. By establishing an effective fusion of multimodal features of block style, a fine-grained style measurement technology method suitable for interpreting the historical and cultural connotations of blocks is obtained, and an automated measurement technology system that is highly aligned with human subjective perception is established, significantly improving the dimensional integrity of block style measurement and the ability to analyze street view connotations. It can not only avoid the "sameness of thousands of cities", but also provide quantifiable and explainable intelligent decision-making support, further explore the highly recognizable cultural symbols of various places, and solve the core pain points of low efficiency, semantic fragmentation and poor interpretability of traditional methods, so as to build urban and rural landscapes with local characteristics.

[0155] According to an embodiment of the present invention, the street view description text obtaining module includes: an element triple submodule, a feature tree determination submodule and a text output submodule.

[0156] The feature triplet submodule is used to process street view images based on the target detection method to obtain multiple feature triplets. The feature triplet represents the feature type, bounding box coordinates and confidence level. The street view image includes features of multiple feature types.

[0157] The feature tree determination submodule is used to process the street view image according to the feature triples and determine the feature tree.

[0158] The text output submodule is used to input the input sequence obtained based on the feature tree into the large language model and output the street scene description text.

[0159] According to an embodiment of the present invention, the feature tree determination submodule includes: a feature interception result unit, a feature attribute information feature unit and a feature tree determination unit.

[0160] The feature interception result unit is used to intercept the street view image according to the feature triples to obtain the feature interception result.

[0161] The feature attribute information feature unit is used to process the feature interception result based on the target detection model to obtain the feature attribute information feature. The feature attribute information feature includes the morphological feature, color feature and material feature of the target feature. The feature interception result includes the target feature.

[0162] The feature tree determination unit is used to determine the feature tree according to the feature attribute information characteristics and the bounding box coordinates.

[0163] According to an embodiment of the present invention, the feature tree determination unit includes: a node subunit and a feature tree determination subunit.

[0164] The node subunit is used to use the street view image as the root node of the feature tree, the feature type in the street view image as the first-level child node, and the feature attribute information features, bounding box coordinates and feature type identifier corresponding to the feature type as the second-level child node.

[0165] The feature tree determining subunit is used to determine the feature tree according to the root node, the first-level child nodes and the second-level child nodes.

[0166] According to an embodiment of the present invention, the text output submodule includes: a bounding box coordinate determination unit, a spatial relationship feature unit, and an input sequence determination unit.

[0167] The bounding box coordinate determination unit is used to determine the bounding box coordinates corresponding to each of the multiple feature types in the feature tree.

[0168] The spatial relationship feature unit is used to determine the feature spatial relationship feature based on multiple bounding box coordinates. The feature spatial relationship feature represents the spatial relationship between feature types. The spatial relationship includes covering, combination and mosaic.

[0169] The input sequence determination unit is used to determine the input sequence according to the feature type, feature attribute information characteristics and feature spatial relationship characteristics.

[0170] According to an embodiment of the present invention, the street view semantic feature determination module includes: a target street view description text acquisition submodule, a vocabulary list acquisition submodule and a street view semantic feature acquisition submodule.

[0171] The target street view description text obtaining submodule is used to perform semantic preprocessing on the street view description text to obtain the target street view description text.

[0172] The vocabulary list obtaining submodule is used to perform word segmentation processing on the target street scene description text to obtain a vocabulary list.

[0173] The street view semantic feature acquisition submodule is used to extract features of words in the vocabulary list to obtain street view semantic features.

[0174] According to an embodiment of the present invention, a sample block appearance evaluation result corresponding to a sample street view image is obtained, the sample block appearance evaluation result including sample evaluation values ​​of each of a plurality of evaluation dimensions, the sample evaluation values ​​being used to measure the characteristics of each sample element in the sample street view image with respect to the evaluation dimensions; the sample street view image is binary-classified based on the average and variance of the plurality of sample evaluation values ​​to obtain a classification value of the sample street view image; a binary classification label vector of the sample street view image is determined based on the classification value; and an initial evaluation model is trained based on the sample street view image and the binary classification label vector to obtain a trained evaluation model.

[0175] According to an embodiment of the present invention, the element type proportion module includes: a target segmentation result submodule, a calculation submodule and a segmentation image acquisition module.

[0176] The target segmentation result submodule is used to process the street view image using the trained semantic segmentation model to obtain the target segmentation result. The target segmentation result represents the segmentation result of the features with feature types in the street view image.

[0177] The calculation submodule is used to calculate the proportion of feature types based on the target segmentation results.

[0178] The segmentation image acquisition module is used to map the target segmentation result to the street view image to obtain a segmentation image.

[0179] According to an embodiment of the present invention, a street view image includes a block street image and a building image; the street view image is obtained by: obtaining an initial street view image; and preprocessing the initial street view image to obtain the street view image, where the preprocessing includes at least one of adjusting brightness, adjusting contrast, removing noise, and normalizing.

[0180] According to embodiments of the present invention, any multiple modules among the feature type proportion module 610, the street view description text acquisition module 620, the street view semantic feature determination module 630, the fusion module 640, and the output module 650 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present invention, at least one of the feature type proportion module 610, the street view description text acquisition module 620, the street view semantic feature determination module 630, the fusion module 640, and the output module 650 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or implemented in any one of software, hardware, and firmware, or any suitable combination of these. Alternatively, at least one of the feature type proportion module 610, the street view description text acquisition module 620, the street view semantic feature determination module 630, the fusion module 640 and the output module 650 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0181] Figure 7 A block diagram of an electronic device suitable for implementing a method for measuring element features for shaping a streetscape is shown according to an embodiment of the present invention.

[0182] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 702 or programs loaded from a storage unit 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0183] The RAM 703 stores various programs and data required for the operation of the electronic device 700. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 executes the programs in the ROM 702 and / or RAM 703 to perform the various operations of the method flow according to the embodiment of the present invention. It should be noted that the above-mentioned programs may also be stored in one or more memories other than the ROM 702 and RAM 703. The processor 701 may also execute the various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0184] According to an embodiment of the present invention, electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to bus 704. Electronic device 700 may also include one or more of the following components connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or modem. Communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. Removable media 711, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 710 as needed, so that computer programs read from the removable media can be installed into storage section 708 as needed.

[0185] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0186] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above, and / or one or more memories other than ROM 702 and RAM 703.

[0187] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the method for measuring the element characteristics of streetscape shaping provided by the embodiments of the present invention.

[0188] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when the computer program is executed by the processor 701. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0189] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 709, and / or installed from a removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0190] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709 and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-described functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0191] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0194] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for measuring element characteristics for shaping streetscapes, characterized in that: include: Processing the street view image using the trained semantic segmentation model to obtain feature type ratios and a segmentation image, wherein the segmentation image represents a visualization of the street view image mapped with the segmentation result, and the feature type ratio represents the area ratio of features having the feature type in the street view image; The street view image is processed using a large language model to obtain street view description text, including: processing the street view image based on a target detection method to obtain a plurality of feature triplets, wherein the feature triplets represent feature types, bounding box coordinates, and confidence levels, the street view image including a plurality of features of each of the feature types; Performing image interception on the street view image according to the element triples to obtain an element interception result; Processing the feature interception results based on the target detection model to obtain feature attribute information features; Determining a feature tree based on the feature attribute information characteristics and the bounding box coordinates; An input sequence obtained based on the feature tree is input into the large language model to output the street scene description text, wherein the street scene description text includes a description of feature attribute information and a description of feature spatial relationships. The input sequence is obtained in the following manner: Determining bounding box coordinates corresponding to each of the plurality of element types in the feature tree; Determining a feature of element spatial relationship based on the plurality of bounding box coordinates, wherein the feature spatial relationship feature indicates that there is a spatial relationship between the element types, and the spatial relationship includes covering, combining, and mosaicking; Determining the input sequence according to the feature type, the feature attribute information characteristics and the feature spatial relationship characteristics of the features; Performing feature extraction on the street scene description text to obtain street scene semantic features; fusing the street view visual features obtained by extracting features from the element type proportions and the segmented image and the street view semantic features to obtain a multimodal fusion feature; The multimodal fusion features are input into a trained evaluation model, and an evaluation result is output. The evaluation result includes multiple evaluation values ​​for measuring the features of each element in the street view image according to different evaluation dimensions. The evaluation result is used to shape the streetscape with local characteristics.

2. According to the element feature measurement method for block appearance shaping according to claim 1, the element attribute information features include the morphological features, color features and material features of the target elements, and the element interception results include the target elements.

3. The method for measuring element characteristics for shaping streetscapes according to claim 2, characterized in that: The determining of a feature tree according to the feature attribute information and the bounding box coordinates includes: The street view image is used as the root node of the feature tree, the feature type in the street view image is used as the first-level child node, and the feature attribute information features, the bounding box coordinates, and the feature type identifier corresponding to the feature type are used as the second-level child nodes; A feature tree is determined according to the root node, the first-level child nodes, and the second-level child nodes.

4. The method for measuring element characteristics for shaping streetscapes according to claim 1, characterized in that: The feature extraction of the street scene description text to obtain street scene semantic features includes: performing semantic preprocessing on the street scene description text to obtain a target street scene description text; Performing word segmentation processing on the target street scene description text to obtain a vocabulary list; Feature extraction is performed on the words in the vocabulary list to obtain street scene semantic features.

5. The method for measuring element characteristics for shaping streetscapes according to claim 1, characterized in that: The trained evaluation model is trained based on the following steps: Obtaining a sample block appearance evaluation result corresponding to the sample street view image, the sample block appearance evaluation result including sample evaluation values ​​of each of a plurality of evaluation dimensions, the sample evaluation values ​​being used to measure features of each sample element in the sample street view image with respect to the evaluation dimensions; performing binary classification on the sample street view image according to the average value and the variance of the plurality of sample evaluation values ​​to obtain a classification value of the sample street view image; Determining a binary classification label vector of the sample street view image according to the classification value; An initial evaluation model is trained according to the sample street view image and the binary classification label vector to obtain a trained evaluation model.

6. The method for measuring element characteristics for shaping streetscapes according to claim 1, characterized in that: The method of processing the street view image using the trained semantic segmentation model to obtain feature type proportions and segmented images includes: Processing the street view image using the trained semantic segmentation model to obtain a target segmentation result, wherein the target segmentation result represents a segmentation result of elements of the element type in the street view image; Calculating the element type ratio according to the target segmentation result; and The target segmentation result is mapped to the street view image to obtain the segmented image.

7. The method for measuring element characteristics for shaping streetscapes according to claim 1, characterized in that: The street view image includes a block street image and a building image; the street view image is obtained by: Get the initial street view image; The initial street view image is preprocessed to obtain the street view image, where the preprocessing includes at least one of adjusting brightness, adjusting contrast, removing noise, and normalizing.

8. A device for measuring the characteristics of elements used in shaping the streetscape, characterized in that: include: An element type ratio module is configured to process the street view image using the trained semantic segmentation model to obtain an element type ratio and a segmentation image, wherein the segmentation image represents a visualization of the street view image mapped with the segmentation result, and the element type ratio represents the area ratio of elements of the element type in the street view image; a street view description text obtaining module, configured to process the street view image using a large language model to obtain a street view description text, wherein the street view description text includes a description of feature attribute information and a description of feature spatial relationships; A street view semantic feature determination module is used to extract features from the street view description text to obtain street view semantic features; a fusion module, configured to fuse the street view visual features and the street view semantic features obtained by extracting features from the element type proportions and the segmented image to obtain multimodal fusion features; An output module is used to input the multimodal fusion features into a trained evaluation model and output an evaluation result, wherein the evaluation result includes multiple evaluation values ​​for measuring the features of each element in the street view image according to different evaluation dimensions, and the evaluation result is used to shape a streetscape with local characteristics. The element feature measurement device for streetscape shaping is used to execute the method according to any one of claims 1 to 7.

Images