Method for quantifying sentiment perception considering scene semantic information based on graph structure

Abstract

This invention discloses a graph-based method for quantifying sentiment perception while considering scene semantic information. First, a label conversion preprocessing step is performed on the sentiment perception dataset of street view images to provide data for training the sentiment perception model. Then, a scene graph generation model based on an attention mechanism is used for transfer learning to obtain entity and entity relationship representations for sentiment perception. These representations are used to generate node and edge features, and multi-dimensional image features are fused to construct richer graph structure features. Next, a confidence-based filtering rule is applied to filter the features, and a graph convolutional network is used for feature fusion and training based on the filtered features. The advantage of this method is that it structures the sentiment-related features in street view images into highly interpretable entity and relationship features, ultimately enabling the training of a quantitative sentiment perception model. This reveals the connection between urban appearance and sentiment perception, deepens the understanding of the physical appearance of urban streets, and thus assists in urban planning decisions.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence image understanding technology, specifically involving a graph-based method for quantifying emotion perception that takes into account scene semantic information. Background Technology

[0002] Visual perception attributes are closely related to residents' behavior and quality of life. Existing research shows that different visual perception attributes have a significant impact on education, health, and population mobility. Therefore, the importance of clarifying the relationship between the physical appearance of the urban environment and residents' emotional perception types is self-evident. However, the visual elements in the urban environment are complex and numerous, and the interaction mechanism between them and residents' perceptions is not yet clear. Furthermore, existing research methods and models have weak interpretability, making it difficult to provide planners with specific quantitative recommendations at the object scale. Therefore, it is necessary to combine appropriate data sources to explore the impact of visual elements and their objective relationships in urban space on residents' emotional perceptions, and to develop quantitative methods that can accurately reflect the relationship between urban visual elements and residents' emotional perceptions from an objective perspective. This will reveal the relationship between emotional perception types and visual elements, facility attributes, and socio-economic dimensions, providing important input for more refined guidance of urban spatial planning.

[0003] With the rapid development of machine learning and deep learning, researchers have developed a large number of deep learning methods based on street view images. Through questionnaires, manual ratings, or using datasets such as PP2 (MIT Place Pulse 2.0 project), they have explored the influence of visual elements in street view scenes on residents' emotional perception types. However, although these methods can reflect the influence of street view visual elements on residents' urban perception attributes to a certain extent, they have the following two shortcomings and limitations: (1) The urban environment described by street view images is complex and contains many urban elements (such as buildings, trees, roads, etc.). There are spatial and semantic relationships between these elements. The feature extraction methods used by traditional quantitative methods can extract features such as texture and shape from images. However, the city is a whole composed of individual entities, and the objective visual elements are the smallest units that make up the city. This form of feature cannot reflect the contribution of a single element to urban perception, has poor interpretability, and therefore cannot provide quantitative decision-making suggestions at the object scale. (2) Visual perception is A multimodal perception process relies not only on visual information but also on semantic understanding and multimodal feature fusion. Visual objects and their relationships (such as spatial and semantic relationships) in street view imagery can be represented in textual form through semantic descriptions. However, current quantification methods lack the integration of these semantic features and fail to effectively capture the spatial semantic relationships between visual elements. Therefore, the characteristics of the urban environment and the practical needs of urban perception attribute research dictate that quantification methods for visual perception must be more interpretable at the object scale and effectively integrate the multimodal features of objective visual elements and their spatial relationships. However, current quantification methods and deep learning frameworks do not consider the spatial semantic relationships between objective elements in street view imagery, have weak feature interpretability, and lack effective feature fusion strategies, thus failing to meet the current practical needs of quantifying visual perception.

[0004] Scene graphs based on graph structures can represent the entire street view image as a topological structure to abstractly express the high-level spatial semantic information in the image. However, how to integrate the characteristics and advantages of scene graphs into the quantification method of urban scene emotion perception based on street view images is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the technical problems existing in the background art, this invention aims to provide a graph-based sentiment perception quantification method that considers scene semantic information. It preprocesses the score labels of existing street view image sentiment perception datasets by performing label transformation, using this as a data source for training the sentiment perception model. It then utilizes an attention-based scene graph generation model for transfer learning to obtain entity representations and entity relation representations for sentiment perception learning. Node and edge features are generated using these entity and relation representations, and multi-dimensional features are fused to construct node and edge features. A confidence-based filtering rule is used to filter the node and edge features, and a graph convolutional network is used for feature fusion and training based on the filtered features. The advantages of this invention are: it can deconstruct the semantic features related to sentiment perception in street view images into more interpretable entity and entity relation features, representing the entire street view image as a topological structure to abstractly express the high-level spatial semantic information in the image, enabling end-to-end training of the sentiment perception quantification model, revealing the intrinsic connection between urban appearance and sentiment perception, deepening the understanding of the physical appearance of urban streets, and assisting decision-makers in planning.

[0006] To solve the technical problem, the technical solution of the present invention is as follows:

[0007] A graph-based method for quantifying sentiment perception that takes into account scene semantic information, the method comprising:

[0008] S1: For street scene images in the existing emotion perception dataset, extract general visual features and perform position encoding using a backbone network. Then, use the pre-trained weights of the trained scene graph generation model to extract high-level semantic features of the image, and obtain multi-layer feature maps, transferred high-dimensional features, position encoding, and embedding representations of entities and relationships.

[0009] S2: Based on the output of step S1, by extracting and fusing the semantic features of entities and the relational information between entities, the node features and edge features required by the graph convolutional network are generated, that is, the generated entity representation and relational representation.

[0010] S3: Filter and normalize the generated entity features and relation features to construct a high-quality graph structure, ensuring that the features input to the graph convolutional network have high confidence and consistency, thereby obtaining high-confidence node features and edge features, as well as the filtered graph structure;

[0011] S4: Using the filtered graph structure, information is transmitted and aggregated using high-confidence node features and edge features to generate a global representation of the graph. Finally, the generated global representation is input into a fully connected layer and trained using an image dataset with positive and negative sentiment labels. The Sigmoid function is used to perform binary classification prediction of sentiment to achieve efficient perception and quantification of sentiment tendencies in street view images.

[0012] Furthermore, prior to step S1, an existing street view image sentiment perception dataset is used. The dataset contains each image and its corresponding sentiment score label. The continuous sentiment score labels are converted into binary classification labels to obtain an image dataset with positive and negative sentiment labels.

[0013] The specific tag conversion process is as follows:

[0014] Given street view images and labels in an existing emotion perception dataset, a certain image score is labeled S. i L is the lower bound for positive samples, U is the upper bound for negative samples, and μ score and σ score Let y be the mean and standard deviation of all image scores under a certain emotional dimension. i The label for the converted image i.

[0015]

[0016] K = μ score -δ·σ score ,

[0017] U = μ score +δ·σ score

[0018]

[0019] Further, step S1 includes:

[0020] S101: Feature Extraction

[0021] The image is input into the backbone network, and multi-layer feature maps of the image are extracted. The extracted features are: Where H and W are the height and width of the image, respectively, and C is the number of channels;

[0022] S102: Transfer Learning

[0023] The general visual features learned on the ImageNet dataset by the pre-trained model using ResNet as the backbone network are transferred to the target task, enabling the model to extract high-dimensional features from images and perform target recognition more accurately; represented as:

[0024]

[0025] By transferring general scene semantic information and prior information about street scenes from the VisualGenome (VG) dataset to the scene graph generation model, the model can more accurately represent objects and their relationships in street scene images; represented as:

[0026] X VG →XSS

[0027] Where X VG X represents the general scene semantic information transferred from the VG dataset. SS This represents the semantic representation of the scene in the migrated street view data;

[0028] S103: Location Code

[0029] Generate spatial location codes for the extracted feature maps Used to capture spatial information in an image, p is the dimension of the positional encoding, which is generated using sinusoidal positional encoding;

[0030] S104: Generate embedded representation

[0031] The extracted feature map F i and spatial location coding P i The input is fed into a Transformer-based scene graph generation model to generate an embedding representation X for each entity and relation. entities and r relations .

[0032] Further, step S2 includes:

[0033] Multi-dimensional features are extracted and fused from the output of the scene graph generation model to generate the node and edge features required by the graph convolutional network, providing semantic, spatial and relational information for subsequent graph structure modeling;

[0034] By integrating the abstract semantic features output by the decoder, the bounding box coordinate regression results, and the object category probability, multidimensional node features are constructed; at the same time, edge features are generated based on relation type prediction, confidence estimation, and the relative differences between the subject and object spaces, providing a standardized input containing semantic, positional, and relational information for subsequent graph structure modeling.

[0035] The entity representation is generated as follows:

[0036]

[0037] Generation of relational representations:

[0038]

[0039] in, d represents the abstract semantic features output by the last layer of the decoder. model For feature dimension, The bounding box coordinates are normalized, including center coordinates and width and height. Let C be the probability distribution of object categories, and C be the number of categories; in the edge features Index for discrete relation types. For relationship confidence, and These are the center coordinate difference between the subject and object and their aspect ratio, which, when spliced ​​together, form a 4-dimensional spatial relative feature.

[0040] Furthermore, step S3 includes:

[0041] S301: Feature Filtering

[0042] Using the node confidence threshold τ node And edge confidence threshold τ edge The system filters out entities and relationships with high confidence; retains entities whose node category probability is higher than the threshold, removes low-confidence edges, reduces noise introduced by false detections or ambiguous relationships, and ensures that the features of the input graph convolutional network have high reliability, providing a data foundation for subsequent semantic fusion.

[0043] S302: Feature Normalization

[0044] The bounding box coordinates are scaled to the [0,1] range according to the image size B to eliminate the influence of image resolution differences on spatial localization; the aspect ratio of the subject and object is logarithmically processed to alleviate the numerical instability caused by extreme size differences; the normalization operation unifies the feature units, enhances the model's sensitivity to spatial features, accelerates the training convergence process, and improves the stability of the classification task.

[0045] Furthermore, step S4 includes:

[0046] S401: Input to the graph neural network model

[0047] The input to the graph neural network is the filtered node features. filtered and edge features filtered ;

[0048] S402: Message Passing and Feature Aggregation

[0049] Node features are updated through a message passing mechanism. Each node aggregates information from its neighboring nodes and updates its features through a graph neural network.

[0050] S403: Graph-level Feature Generation

[0051] Global average pooling is used to aggregate node features to obtain a global representation of the graph.

[0052] S404: Binary Classification Task Modeling

[0053] Graph-level features are input into a fully connected layer and mapped to sentiment binary classification probabilities using a Sigmoid function. in The closer to 1, the higher the probability of a positive sample in the image; the closer to 0, the lower the probability of a positive sample. The classification result directly reflects the overall sentiment of the street view image samples.

[0054] S405: Loss Function and Training

[0055] By training each model separately, i.e., training the model independently on different sentiment dimensions (such as the six sentiment dimensions of the PP2 dataset), and using the binary cross-entropy loss function to optimize the model, where y i ∈{0,1} represents the true sentiment label in the corresponding dimension; the weights and classification parameters of the graph convolutional network layer are adjusted end-to-end through backpropagation to suppress the interference of low-confidence features on classification, enhance the model's ability to perceive complex street scene semantics, and gradually strengthen the mapping relationship between graph structure features and sentiment labels during training, ultimately achieving high-precision classification of street scene images in various sentiment dimensions.

[0056] A graph-structured sentiment perception quantization system that considers scene semantic information, the system being used to execute any of the methods described above, the system comprising:

[0057] Emotion-aware street view image dataset preprocessing module: Using an existing street view image emotion-aware dataset, the dataset needs to contain each image and its corresponding emotion score label. The continuous emotion score labels are converted into binary labels to obtain an image dataset with positive and negative emotion labels.

[0058] High-dimensional feature extraction module: For street scene images in the existing sentiment perception dataset, the backbone network is used to extract general visual features and perform position encoding. Then, the pre-trained weights of the trained scene graph generation model are used to extract high-level semantic features of the image, resulting in multi-layer feature maps, transferred high-dimensional features, position encoding, and embedding representations of entities and relationships.

[0059] Scene graph generation module: Based on the output of the high-dimensional feature extraction module, it extracts and fuses the semantic features of entities and the relational information between entities to generate the node features and edge features required by the graph convolutional network, that is, the generated entity representation and relational representation;

[0060] Street view entity prediction module: It filters and normalizes the generated entity features and relationship features, constructs a high-quality graph structure, and ensures that the features input to the graph convolutional network have high confidence and consistency, thereby obtaining high-confidence node features and edge features, as well as the filtered graph structure;

[0061] Street view entity relationship prediction module: Through the filtered graph structure, information is transmitted and aggregated using high-confidence node features and edge features to generate a global representation of the graph. Finally, the generated global representation is input into a fully connected layer and trained using an image dataset with positive and negative sentiment labels. The Sigmoid function is used to perform binary classification prediction of sentiment to achieve efficient perception and quantification of sentiment tendencies in street view images.

[0062] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements an emotion perception quantification method based on graph structure that takes into account scene semantic information, as described above.

[0063] A computer-readable storage medium storing a computer program that, when executed by a processor, implements, as described above, a graph-structure-based method for quantifying emotion perception that takes into account scene semantic information.

[0064] Compared with the prior art, the advantages of the present invention are as follows:

[0065] First, a street view dataset with positive and negative sentiment labels is used to ensure that the model obtains a real and reliable sentiment standard during training, thereby improving the accuracy and reliability of predictions.

[0066] Secondly, the solution utilizes a pre-trained backbone network and scene graph generation model to extract multi-layered visual features and high-level semantic information. This deep feature learning capability enables the model to capture emotional features in complex street scene images effectively.

[0067] By using graph convolutional networks, the approach effectively models the relationships between nodes, fully mining the semantic and spatial information of entities, thereby enhancing the performance of sentiment classification. Simultaneously, feature selection and normalization improve the confidence of input features, significantly reduce the impact of noise, and enhance the robustness of the model.

[0068] The flexibility and scalability of this approach enable it to adapt to analyses across different sentiment dimensions, facilitating subsequent secondary development. Furthermore, the explicit feature extraction and graph construction processes enhance the model's interpretability, allowing users to understand the basis for sentiment predictions and increasing user trust.

[0069] In summary, this invention constructs an efficient, accurate, and highly interpretable emotion perception model, applicable to various application scenarios in modern smart cities and transportation systems. Attached Figure Description

[0070] Figure 1 The main flowchart of the emotion perception quantification method based on graph structure that takes into account scene semantic information in this invention. Detailed Implementation

[0071] The specific implementation of the present invention is described below with reference to embodiments:

[0072] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0073] Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity of description and are not intended to limit the scope of the invention. Any changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0074] Example 1:

[0075] like Figure 1 As shown, the present invention provides a method and system for quantifying emotion perception that takes into account scene semantic information based on graph structure, including the following steps:

[0076] S1. Prepare a dataset (street view format) that can represent the emotional perception status of a certain area, as the data source for training the emotional perception model. Then, process the score labels of the dataset and convert them into positive and negative sample labels for binary classification tasks.

[0077] S2. Using the dataset in S1 as input data, extract general visual features and perform position encoding using the backbone network. Use the trained scene graph generation model to extract high-level features of each image in the dataset and generate entity representations and relationships between entities.

[0078] S3. Select the intermediate features and prediction results from the previous step to generate node features and edge features, and fuse multi-dimensional features to construct node features and edge features.

[0079] S4. Set filtering rules to filter node features and edge features to ensure that the subsequent graph convolutional neural network receives high-quality training features.

[0080] S5. Using the graph structure information generated in S4 as input, a graph convolutional neural network is used to fuse the visual semantic information of the image to perform the classification task of street scene emotion perception images.

[0081] The specific label conversion operations in S1 are as follows:

[0082] Given street view images and labels in an existing emotion perception dataset, a certain image score is labeled S. i L is the lower bound for positive samples, U is the upper bound for negative samples, and μ score and σ score Let y be the mean and standard deviation of all image scores under a certain emotional dimension. i The label for the converted image i.

[0083]

[0084] K = μ score -δ·σ score ,

[0085] U = μ score +δ·σ score

[0086]

[0087] S2 is explained in detail below:

[0088] Select backbone network

[0089] To capture multi-scale visual information in images, ResNet50 was chosen as the backbone network. ResNet50, with its residual connection structure, effectively mitigates the vanishing gradient problem in deep networks, while simultaneously extracting hierarchical features from local details to global semantics through stacked convolutional layers.

[0090] The image is input into the backbone network, and ResNet50 is used to extract multi-layer feature maps of the image. The extracted image features are... Where H and W are the height and width of the image, respectively, and C is the number of channels.

[0091] Deep features of ResNet50 F i It contains rich semantic information (such as object category and texture) and spatial information (such as outline position), providing a high-quality visual feature foundation for subsequent Transformers.

[0092] Transfer learning

[0093] The core of transfer learning lies in using the general knowledge of pre-trained models to accelerate the convergence of the target task and improve the model's generalization ability. In the system and method described in this invention, two types of transfer learning are involved in the steps.

[0094] (1) Visual Feature Transfer: The general visual features learned by the ResNet50 pre-trained model on the ImageNet dataset are transferred to the target task, enabling the model to extract high-dimensional features from images more accurately and perform target recognition. The formulaic expression of transfer learning is as follows:

[0095]

[0096] in This represents image features transferred from the ImageNet dataset. This represents the image features in the target task data after migration.

[0097] (2) Semantic knowledge transfer: General scene semantic information and prior information about street scenes from the VisualGenome (VG) dataset are transferred to the scene graph generation model, enabling the model to more accurately represent objects and their relationships in street scene images. The formulaic representation of transfer learning is as follows:

[0098]

[0099] in This represents the entity feature representation transferred from the VG dataset. This represents the entity feature representation in the migrated street view data.

[0100] Location coding

[0101] Convolutional neural networks lack awareness of pixel location information in images, while object detection tasks heavily rely on spatial information. Positional encoding, by explicitly injecting coordinate information, helps the model distinguish objects at different locations and model spatial relationships.

[0102] Image pixel position codes are generated using sine encoding and learned position coding, converting the pixel image into a high-dimensional vector containing the image's spatial information.

[0103] In the sinusoidal position encoding, sine and cosine functions are used to encode each position x and y, respectively:

[0104]

[0105] Where, x embed and y embed These are the horizontal and vertical spatial coordinates of the image, dim. t It is the encoding scale generated on the feature dimension.

[0106] Similarly, learned positional coding can also be used to generate positional codes by embedding each position of the image into a trained vector space.

[0107] P = rowembed(i) + colembed(j)

[0108] Where i is the column coordinate, j is the row coordinate, and rowembed(i) and colembed(j) represent the row and column embedding vectors, respectively. The generated spatial location code... With image features F i Combined, this provides spatial information for each image location.

[0109] Furthermore, the feature location-aware information generated by merging spatial location encoding and image features can be represented as: Inputfeatures = F i +P i After adjusting the number of channels to adapt to the input dimension of the Transformer, this feature is input into the Transformer encoder. Through a self-attention mechanism, it processes feature location-aware information, learns the relationships between different locations in the image, and transforms the local features extracted from the backbone into a globally context-aware feature representation. This fuses semantic and spatial information, providing a rich contextual foundation for the decoder. The encoder's computation process can be represented as follows:

[0110] Q,K,V = Linear(inputfeatures)

[0111]

[0112] Here, Q, K, and V represent the query, key, and value, respectively, which are obtained from the input features through linear transformation. In this way, the Transformer encoder can capture long-range dependencies between image features.

[0113] After processing by the decoder, these high-dimensional feature representations are decoded into more specific embedding representations. Specifically, the decoder's input includes not only the output src (local abstract features) from the encoder, but also the positional encoding P generated earlier. target This is used to tell the decoder the specific location of the target in space.

[0114] Input target embedding X in decoder target Self-attention layer computation of self-attention mechanism:

[0115] Q target ,K target V target =Linear(X) target )

[0116] The output is:

[0117]

[0118] Next, a cross-attention layer is used, taking the image features X from the encoder as input. image and target embedding X target Calculate cross-attention:

[0119] Q target ,K image V image =Linear(X) target ),Linear(X image )

[0120] The output is:

[0121]

[0122] Feedforward Neural Network (FFN):

[0123] Apply a feedforward neural network (FFN) transformation to the position of each target:

[0124] FFN(X target ) = max(0,X target W1+b1)W2+b2

[0125] Layer normalization is applied to the outputs of self-attention and cross-attention:

[0126]

[0127] After processing by the decoder, the multi-layer output hs (HiddenStates) of the entity decoder and the multi-layer output hs of the triplet decoder are obtained. t (Hidden States triplets) predict entity categories through a linear layer, and a multilayer perceptron generates the coordinates of a four-dimensional bounding box, which are then normalized to [0,1] using a sigmoid function; the entity decoder outputs... Multi-layer decoding results are used to predict object categories and bounding boxes. Triple decoder output. After splitting, independent representations of the subject and object are obtained, which are used for relation prediction.

[0128] Furthermore, suitable features generated by the Transformer and the prediction results of the prediction head are selected as the source of feature construction for subsequent node features and edge features, providing high-quality training data input for subsequent graph convolutional neural networks.

[0129] S3 is explained in detail below:

[0130] S31 Node Feature Generation

[0131] Node features are used to represent detected entities (objects) and consist of the following three parts, forming highly semantic and spatially aware node features:

[0132] (1) The output hs of the last layer of the decoder is used as the abstract feature nodeshs of the entity, and is represented as:

[0133]

[0134] Where B is the batch size, N q d is the preset number of queries. model hs[-1] is the feature dimension, and hs[-1] is the output of the last layer of the decoder, which contains the abstract semantic information of the last layer.

[0135] (2) Bounding box coordinates: Using the bounding box regression head, the coordinates of the bounding boxes are generated after Sigmoid normalization, providing the positional feature information of the entities, represented as:

[0136]

[0137] Where σ is the Sigmoid function, outputting the normalized bounding box coordinates [c x ,c y ,w,h], This is the weight matrix of the bounding box regression head.

[0138] (3) Class Probabilities: Using the regression head of class probabilities, after Softmax processing, the object class probabilities nodeclass probs are obtained, expressed as:

[0139]

[0140] Where C is the number of object categories, excluding the background category. This is the weight matrix for the classification heads.

[0141] After concatenating the above features, information such as "what" (category), "where" (coordinates), and "how" the object is related is simultaneously encoded, providing multi-dimensional information for the graph convolutional network to obtain the final node feature matrix, which can be represented in the following form:

[0142]

[0143] S32 edge feature generation

[0144] Edge features are used to describe the relationships between entities, and need to characterize the relationship type, confidence level, and spatial interaction pattern between entities. The reasoning of complex relationships by the supporting graph network consists of the following three parts:

[0145] (1) Relationship type: The prediction result using the relation classification head is represented as:

[0146]

[0147] Where N r The number of triples, where each value is the predicted relation category index. This is the raw output of the relation classification header, where R is the number of relation categories.

[0148] (2) Relationship confidence:

[0149] Represented as:

[0150]

[0151] (3) Spatial relative characteristics (referring to the differences and distinctions between the subject and object in space, used to represent the differences in the relational characteristics between the subject and object):

[0152] The calculation based on the bounding box of the subject and object can be represented as:

[0153]

[0154] The above features are concatenated into edge features, represented as follows:

[0155]

[0156] Furthermore, S4 is explained in detail below:

[0157] S41 Feature Filtering:

[0158] (1) Node feature filtering:

[0159] Retaining high-confidence entity features, and setting a confidence threshold nodeτ, the node feature selection process can be represented as follows:

[0160] validnoses=nodefeatures[max(node ​​class probs,dim=-1)>τ node ]

[0161] (2) Edge feature selection:

[0162] Retaining high-confidence relationship features, and setting a confidence threshold edgeτ, the edge feature selection process can be represented as:

[0163] valid edges=edge features[edge conf>τ edge ]

[0164] S42 feature normalization:

[0165] Noise removal detection reduces computational redundancy in graph convolutional networks; normalization ensures balance across different feature scales, accelerating model convergence.

[0166] (1) Coordinate normalization

[0167] (S is the input image size)

[0168] (2) Logarithmic aspect ratio

[0169] wh ratio = log(wh ratio + 1)

[0170] Furthermore, a detailed description of S5 follows:

[0171] S51: Based on the node and edge features filtered in S4, construct the graph structure information. Define graph G = (V, E), where:

[0172] Node set V: Each node corresponds to a valid entity, and the node features are those of the filtered entities. Where N v d represents the number of valid nodes. model 4 represents the abstract feature dimension output by the decoder, 4 represents the normalized coordinates, and C represents the number of object categories.

[0173] Edge set E: Each edge corresponds to a filtered valid relation, and the edge characteristics are... Where N e To determine the number of effective edges, the 6-dimensional features include relation type, confidence level, and spatial relative features (Δ). xy wh ratio ).

[0174] Adjacency Matrix Based on edge features, if there is an edge between nodes i and j, then A ij =edge conf ×onehot(edgetype) represents the connection strength using the joint weight of relation confidence and type encoding.

[0175] S52 Message Passing and Feature Aggregation

[0176] By aggregating neighborhood information through multi-layer graph convolution operations and fusing node and edge features, a high-order graph representation is generated. The specific steps are as follows:

[0177] Message passing:

[0178] When a graph convolutional neural network updates node features, this formula can be used to illustrate the process:

[0179]

[0180] Among these:

[0181] The adjacency matrix for adding self-loops, where I is the identity matrix. for The degree matrix, For the features of the l-th layer nodes, the initial input H (0) =nodefeatures. Let be the trainable weight matrix. σ is the activation function.

[0182] Multi-layer feature stacking:

[0183] Stacked LL-layer GCNs capture multi-hop neighborhood information, and the final output node features are:

[0184]

[0185] S53 Graph-level Feature Generation

[0186]

[0187] S54 binary classification task modeling

[0188] The graph-level features are input into a fully connected layer and a Sigmoid function, and the output is the sentiment perception classification probability.

[0189]

[0190] b cls ∈R, where R is the classification layer parameter, representing the probability that the image belongs to the "positive sentiment" category. This indicates the probability that the image belongs to the "positive emotion" category.

[0191] S55 Loss Function and Training

[0192] The model is optimized using a binary cross-entropy loss function:

[0193]

[0194] Where y i ∈{0,1} represents the true label, and the model updates the GCN weights {W} through backpropagation. (l) Classification layer parameters W cls b cls This enables end-to-end training.

[0195] Example 2:

[0196] This invention provides a graph-structured sentiment perception quantization system that considers scene semantic information. This system can be used to implement the aforementioned graph-structured sentiment perception quantization method that considers scene semantic information. Specifically, it includes:

[0197] The emotion-aware street view image dataset preprocessing module: Based on the street view images and corresponding score labels in the original dataset, the module filters out noisy data and sets thresholds to convert the original score labels of the street view images into binary classification labels.

[0198] High-dimensional feature extraction module: Extracts general high-dimensional visual features from street view images.

[0199] The scene graph generation module includes the following sub-modules:

[0200] This module takes street view images from the dataset as input, processes the input street view images using a high-dimensional feature extraction module, and uses the generated high-dimensional visual features combined with an attention mechanism to predict scene entities and the relationships between entities.

[0201] Street View Entity Prediction Module: Uses an attention mechanism combined with an encoder and entity decoder to predict street view entities.

[0202] Street View Entity Relationship Prediction Module: Uses an attention mechanism combined with an encoder and a triple decoder to predict the relationships between street view entities.

[0203] Graph-level feature integration and generation module: Combines the intermediate abstract semantic features and prediction results from the scene graph generation module to generate graph-level features for training the sentiment perception graph convolutional network.

[0204] Graph Convolutional Sentiment Semantic Learning Module:

[0205] The graph-level features generated by the scene graph generation module are used for feature aggregation and message passing, and a binary classification task of positive and negative target sentiment samples is performed until the training and learning of sentiment perception scene semantics is completed.

[0206] Example 3:

[0207] This embodiment provides a terminal device, which includes a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve corresponding method flows or corresponding functions. The processor described in this embodiment can be used for the operation of an emotion perception quantization method based on graph structure and considering scene semantic information, including the following steps:

[0208] S1: For street scene images in the existing emotion perception dataset, extract general visual features and perform position encoding using a backbone network. Then, use the pre-trained weights of the trained scene graph generation model to extract high-level semantic features of the image, and obtain multi-layer feature maps, transferred high-dimensional features, position encoding, and embedding representations of entities and relationships.

[0209] S2: Based on the output of step S1, by extracting and fusing the semantic features of entities and the relational information between entities, the node features and edge features required by the graph convolutional network are generated, that is, the generated entity representation and relational representation.

[0210] S3: Filter and normalize the generated entity features and relation features to construct a high-quality graph structure, ensuring that the features input to the graph convolutional network have high confidence and consistency, thereby obtaining high-confidence node features and edge features, as well as the filtered graph structure;

[0211] S4: Using the filtered graph structure, information is transmitted and aggregated using high-confidence node features and edge features to generate a global representation of the graph. Finally, the generated global representation is input into a fully connected layer and trained using an image dataset with positive and negative sentiment labels. The Sigmoid function is used to perform binary classification prediction of sentiment to achieve efficient perception and quantification of sentiment tendencies in street view images.

[0212] Example 4:

[0213] This embodiment provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0214] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the emotion perception quantification method based on graph structure and considering scene semantic information in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps:

[0215] S1: For street scene images in the existing emotion perception dataset, extract general visual features and perform position encoding using a backbone network. Then, use the pre-trained weights of the trained scene graph generation model to extract high-level semantic features of the image, and obtain multi-layer feature maps, transferred high-dimensional features, position encoding, and embedding representations of entities and relationships.

[0216] S2: Based on the output of step S1, by extracting and fusing the semantic features of entities and the relational information between entities, the node features and edge features required by the graph convolutional network are generated, that is, the generated entity representation and relational representation.

[0217] S3: Filter and normalize the generated entity features and relation features to construct a high-quality graph structure, ensuring that the features input to the graph convolutional network have high confidence and consistency, thereby obtaining high-confidence node features and edge features, as well as the filtered graph structure;

[0218] S4: Using the filtered graph structure, information is transmitted and aggregated using high-confidence node features and edge features to generate a global representation of the graph. Finally, the generated global representation is input into a fully connected layer and trained using an image dataset with positive and negative sentiment labels. The Sigmoid function is used to perform binary classification prediction of sentiment to achieve efficient perception and quantification of sentiment tendencies in street view images.

[0219] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0220] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0221] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0222] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0223] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

[0224] Many other changes and modifications can be made without departing from the concept and scope of this invention. It should be understood that this invention is not limited to the specific embodiments, and the scope of this invention is defined by the appended claims.

Claims

1. A graph-based method for quantifying sentiment perception that takes into account scene semantic information, characterized in that, The method includes: S1: For street scene images in the existing emotion perception dataset, extract general visual features and perform position encoding using a backbone network. Then, use the pre-trained weights of the trained scene graph generation model to extract high-level semantic features of the image, and obtain multi-layer feature maps, transferred high-dimensional features, position encoding, and embedding representations of entities and relationships. S2: Based on the output of step S1, by extracting and fusing the semantic features of entities and the relational information between entities, the node features and edge features required by the graph convolutional network are generated, that is, the generated entity representation and relational representation. S3: Filter and normalize the generated entity features and relation features to construct a high-quality graph structure, ensuring that the features input to the graph convolutional network have high confidence and consistency, thereby obtaining high-confidence node features and edge features, as well as the filtered graph structure; S4: Using the filtered graph structure, information is transmitted and aggregated using high-confidence node features and edge features to generate a global representation of the graph. Finally, the generated global representation is input into a fully connected layer and trained using an image dataset with positive and negative sentiment labels. The Sigmoid function is used to perform binary classification prediction of sentiment to achieve efficient perception and quantification of sentiment tendencies in street view images. Step S1 includes: S101: Feature extraction; The image is input into the backbone network, and multi-layer feature maps of the image are extracted. The extracted features are: ,in and These are the height and width of the image, respectively. It is the number of channels; S102: Transfer learning; The general visual features learned on the ImageNet dataset by the pre-trained model using ResNet as the backbone network are transferred to the target task, enabling the model to extract high-dimensional features from images and perform target recognition more accurately; represented as: ， By transferring general scene semantic information and prior information about street scenes from the Visual Genome dataset to the scene graph generation model, the model can more accurately represent objects and their relationships in street scene images; represented as: ， in This represents general scene semantic information migrated from the Visual Genome dataset. This represents the semantic representation of the scene in the migrated street view data; S103: Location code; Generate spatial location codes for the extracted feature maps Used to capture spatial information in images. It is the dimension of the positional encoding, which is generated using sinusoidal positional encoding; S104: Generate embedded representation; Extracted feature maps and spatial location coding The input is fed into a Transformer-based scene graph generation model to generate an embedded representation of each entity and relation. and .

2. The sentiment perception quantification method based on graph structure that takes into account scene semantic information as described in claim 1, characterized in that, Before step S1, an existing street view image sentiment perception dataset is used. The dataset contains each image and its corresponding sentiment score label. The continuous sentiment score labels are converted into binary labels to obtain an image dataset with positive and negative sentiment labels.

3. The emotion perception quantification method based on graph structure that takes into account scene semantic information as described in claim 2, characterized in that, The specific tag conversion process is as follows: Given street view images and labels in an existing emotion perception dataset, a certain image score is labeled as... , The lower bound for positive samples. This is the upper bound for negative samples. and Let be the mean and standard deviation of all image scores under a certain emotional dimension. For the converted image Tags; ， ， ， 。 4. The sentiment perception quantification method based on graph structure that takes into account scene semantic information as described in claim 1, characterized in that, Step S2 includes: Multi-dimensional features are extracted and fused from the output of the scene graph generation model to generate the node and edge features required by the graph convolutional network, providing semantic, spatial and relational information for subsequent graph structure modeling; By integrating the abstract semantic features output by the decoder, the bounding box coordinate regression results, and the object category probability, multidimensional node features are constructed; at the same time, edge features are generated based on relation type prediction, confidence estimation, and the relative differences between the subject and object spaces, providing a standardized input containing semantic, positional, and relational information for subsequent graph structure modeling. The entity representation is generated as follows: ， Generation of relational representations: ， in, This represents the abstract semantic features output by the last layer of the decoder. For feature dimension, The bounding box coordinates are normalized, including center coordinates and width and height. For object category probability distribution, Number of categories; in edge features Index for discrete relation types. For relationship confidence, These are the center coordinate difference between the subject and object and their aspect ratio, which, when spliced ​​together, form a 4-dimensional spatial relative feature.

5. The sentiment perception quantification method based on graph structure that takes into account scene semantic information according to claim 1, characterized in that, Step S3 includes: S301: Feature Filtering Using node confidence thresholds And edge confidence threshold The system filters out entities and relationships with high confidence; retains entities whose node category probability is higher than the threshold, removes low-confidence edges, reduces noise introduced by false detections or ambiguous relationships, and ensures that the features of the input graph convolutional network have high reliability, providing a data foundation for subsequent semantic fusion. S302: Feature Normalization Adjust bounding box coordinates according to image size Zoom to The interval is used to eliminate the impact of image resolution differences on spatial localization; the aspect ratio of the subject and object is logarithmically processed to alleviate the numerical instability caused by extreme size differences; the normalization operation unifies the feature dimensions, enhances the model's sensitivity to spatial features, accelerates the training convergence process, and improves the stability of the classification task.

6. The sentiment perception quantification method based on graph structure that takes into account scene semantic information according to claim 1, characterized in that, Step S4 includes: S401: Input to the graph neural network model The input to the graph neural network is the filtered node features. and edge features ; S402: Message Passing and Feature Aggregation Node features are updated through a message passing mechanism. Each node aggregates information from its neighboring nodes and updates its features through a graph neural network. S403: Graph-level Feature Generation Global average pooling is used to aggregate node features to obtain a global representation of the graph. ; S404: Binary Classification Task Modeling The graph-level features are input into the fully connected layer and mapped to the sentiment binary classification probability ŷ∈[0,1] by the Sigmoid function; where the closer ŷ is to 1, the higher the probability of positive samples in the image, and the closer it is to 0, the lower the probability of positive samples. The classification result directly reflects the overall sentiment tendency of the street scene image samples. S405: Loss Function and Training By training each model separately—that is, training different sentiment dimensions independently—the model is optimized using a binary cross-entropy loss function. ∈{0,1} represents the true sentiment label in the corresponding dimension; the weights and classification parameters of the graph convolutional network layer are adjusted end-to-end through backpropagation to suppress the interference of low-confidence features on classification, enhance the model's ability to perceive complex street scene semantics, and gradually strengthen the mapping relationship between graph structure features and sentiment labels during training, ultimately achieving high-precision classification of street scene images in various sentiment dimensions.

7. A graph-based emotion perception quantification system that takes into account scene semantic information, characterized in that, The system is used to perform the method according to any one of claims 1-6, the system comprising: The preprocessing module for the emotion-aware street view image dataset uses an existing street view image emotion-aware dataset. The dataset contains each image and its corresponding emotion score label. The continuous emotion score labels are converted into binary labels to obtain an image dataset with positive and negative emotion labels. High-dimensional feature extraction module: For street scene images in the existing sentiment perception dataset, the backbone network is used to extract general visual features and perform position encoding. Then, the pre-trained weights of the trained scene graph generation model are used to extract high-level semantic features of the image, resulting in multi-layer feature maps, transferred high-dimensional features, position encoding, and embedding representations of entities and relationships. Scene graph generation module: Based on the output of the high-dimensional feature extraction module, it extracts and fuses the semantic features of entities and the relational information between entities to generate the node features and edge features required by the graph convolutional network, that is, the generated entity representation and relational representation; Street view entity prediction module: It filters and normalizes the generated entity features and relationship features, constructs a high-quality graph structure, and ensures that the features input to the graph convolutional network have high confidence and consistency, thereby obtaining high-confidence node features and edge features, as well as the filtered graph structure; Street view entity relationship prediction module: Through the filtered graph structure, information is transmitted and aggregated using high-confidence node features and edge features to generate a global representation of the graph. Finally, the generated global representation is input into a fully connected layer and trained using an image dataset with positive and negative sentiment labels. The Sigmoid function is used to perform binary classification prediction of sentiment to achieve efficient perception and quantification of sentiment tendencies in street view images.

8. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the emotion perception quantification method based on graph structure that takes into account scene semantic information, as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the emotion perception quantification method based on graph structure and considering scene semantic information, as described in any one of claims 1 to 6.

Images