Scene semantic information-considered emotion perception quantification method based on graph structure

Abstract

The invention discloses an emotion perception quantification method considering scene semantic information based on a graph structure, which comprises the following steps of: firstly, performing label conversion preprocessing on an emotion perception data set of a streetscape image, and providing a data source for training an emotion perception model; and carrying out transfer learning by adopting a scene graph generation model based on an attention mechanism to obtain entity representation and entity relationship representation for emotion perception. Through the representations, node features and edge features are generated, and multi-dimensional image features are fused to construct richer graph structure features. Then, screening the features by applying a confidence screening rule, and carrying out feature fusion and training by using a graph convolutional network based on the screened features; the method has the advantages that the emotion-related features in the streetscape image are structured into highly explained entity and relation features, and finally quantitative model training of emotion perception is realized to reveal the relation between the city appearance and the emotion perception, deepen the understanding of the physical appearance of the city street and further assist in decision making of city planning.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence image understanding, and particularly relates to a sentiment perception quantification method based on a graph structure and considering scene semantic information. BACKGROUND

[0002] Visual perception attributes are closely related to the behavior and quality of life of residents. Existing research has shown that different visual perception attributes have an undeniable impact on education, health, and population mobility, so the importance of clarifying the correlation between the physical appearance of the urban environment and the types of residents' emotional perception is self-evident. However, the visual elements in the urban environment are complex and numerous, and the interaction mechanism between them and the residents' perception is not clear. At the same time, the existing research methods and models have weak interpretability, making it difficult to give planners specific quantitative suggestions at the object scale. Therefore, it is necessary to explore the influence of visual elements and their objective relationships in urban space on residents' emotional perception by combining appropriate data sources, and to develop a quantitative method that can accurately reflect the relationship between urban visual elements and residents' emotional perception from an objective level, in order to reveal the correlation between emotional perception types, visual elements, facility attributes, and socio-economic dimensions, and to provide important input for more fine urban space planning.

[0003] With the rapid development of machine learning and deep learning, researchers have developed a large number of deep learning methods based on street view images, and through questionnaire survey, artificial scoring or using PP2 (MIT Place Pulse 2.0 project) and other data sets, the influence of visual elements in street view scenes on the perception type of residents' emotions is explored. However, these methods can reflect the influence of street view visual elements on the perception of residents' urban attributes to a certain extent, but there are two shortcomings and limitations as follows: (1) The urban environment described by street view images is complex, containing many urban elements (such as buildings, trees, roads, etc.), and there are spatial and semantic relationships between these elements. Although the traditional quantitative method can extract features such as texture and shape from the image, the city is a whole composed of individual entities, and the objective visual elements are the smallest unit of the city. This form of feature cannot reflect the contribution of individual elements to urban perception, and the explainability is poor, which makes it difficult to provide quantitative decision-making suggestions at the object scale. (2) Visual perception is a multi-modal perception process, which not only depends on visual information, but also involves semantic understanding of visual information and multi-modal feature fusion. The visual objects and their relationships (such as spatial relationships and semantic relationships) in street view images can be represented in text form through semantic description, but the current quantitative method lacks the integration of these semantic features, and cannot effectively capture the spatial and semantic relationships between visual elements. Therefore, the characteristics of the above urban environment and the real needs of the study of urban perception attributes determine that the quantitative method of visual perception should not only be more explainable at the object scale, but also well integrate the multi-modal features of objective visual elements and their spatial relationships. However, the current quantitative method and deep learning framework do not consider the spatial and semantic relationships between objective elements in street view images, the feature explainability is weak, and there is a lack of good feature fusion strategy, which cannot meet the real needs of the quantitative visual perception.

[0004] The scene graph based on the graph structure can represent the entire street view image as a topological structure to abstract the high-level spatial semantic information in the image. However, how to integrate the characteristics and advantages of the scene graph into the quantitative method of urban scene emotional perception based on street view images is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] The present application aims to provide a method for quantifying sentiment perception based on graph structure and considering scene semantic information, which converts the score labels of the existing street image sentiment perception dataset for label conversion preprocessing, and uses the converted labels as the data source for training and learning of the sentiment perception model; uses a scene graph generation model based on an attention mechanism for transfer learning to obtain entity representation and entity relationship representation for sentiment perception learning; uses the entity representation and entity relationship representation to generate node features and edge features, and fuses multi-dimensional features to construct node features and edge features; uses a confidence screening rule to screen the node features and edge features, and uses a graph convolution network based on the screened features for feature fusion and training.

[0006] To solve the technical problem, the technical solution of the present application is:

[0007] A method for quantifying sentiment perception based on graph structure and considering scene semantic information, the method comprising:

[0008] S1: For the street image in the existing sentiment perception dataset, a backbone network is used to extract general visual features and perform position encoding, and then the pre-training weight of the trained scene graph generation model is used to extract high-level semantic features of the image to obtain multi-layer feature maps, transferred high-dimensional features, position encoding, and embedded representation of entities and relationships;

[0009] S2: Based on the output of step S1, the node features and edge features required by the graph convolution network are generated by extracting and fusing the semantic features of entities and the relationship information between entities, i.e., the generated entity representation and relationship representation;

[0010] S3: The generated entity features and relationship features are screened and standardized to build a high-quality graph structure, ensuring that the features input into the graph convolution network have high confidence and consistency, thereby obtaining high-confidence node features and edge features, and a screened graph structure;

[0011] S4: Through the screened graph structure, information transmission and aggregation are performed using the high-confidence node features and edge features to generate a global representation of the graph, and finally, the generated global representation is input into a fully connected layer and trained using an image dataset with positive and negative emotion labels, and a Sigmoid function is used for binary classification prediction of emotion to achieve efficient perception and quantification of the sentiment tendency of street images.

[0012] Further, before the step S1, an existing street view image emotion perception dataset is adopted, the dataset contains each image and its corresponding emotion score label, the continuous emotion score label is converted into a binary classification label, and an image dataset with positive and negative emotion labels is obtained;

[0013] The label conversion operation is as follows:

[0014] For the street view images and labels in the existing emotion perception dataset, a certain image score label is S i , L is the lower bound of the positive sample, U is the upper bound of the negative sample, μ score and σ score are the mean and standard deviation of all image scores in a certain emotion dimension, and y i is the label of the converted image i.

[0015]

[0016] K=μ score -δ·σ score ,

[0017] U=μ score +δ·σ score

[0018]

[0019] Further, the step S1 comprises:

[0020] S101: feature extraction

[0021] The image is input into the backbone network, and multi-layer feature maps of the image are extracted, and the extracted features are where H and W are the height and width of the image respectively, and C is the number of channels;

[0022] S102: transfer learning

[0023] The general visual features learned by the ResNet used as the pre-trained model of the backbone network on the ImageNet dataset are transferred to the target task, so that the model can more accurately extract high-dimensional features in the image and perform target recognition; represented as:

[0024]

[0025] The general scene semantic information in the VisualGenome (VG) dataset and the prior information about the street view are transferred to the scene graph generation model, so that the model can more accurately express the objects and their mutual relationships in the street view image; represented as:

[0026] X VG →XSS

[0027] wherein X VG represents the general scene semantic information migrated from the VG dataset, X SS represents the scene semantic representation in the migrated street view data;

[0028] S103: Position encoding

[0029] Generate spatial position encoding for the extracted feature map for capturing spatial information in the image, p is the dimension of the position encoding, and the position encoding is generated by sinusoidal position encoding;

[0030] S104: Generate embedding representation

[0031] Input the extracted feature map F i and the spatial position encoding P i into the scene graph generation model based on the Transformer, to generate the embedding representation X entities and r relations for each entity and relationship.

[0032] Further, the step S2 comprises:

[0033] Extract and fuse multi-dimensional features from the output of the scene graph generation model to generate node features and edge features required by the graph convolutional network, providing semantic, spatial and relationship information for subsequent graph structure modeling;

[0034] By integrating the abstract semantic features, bounding box coordinate regression results and object class probability output by the decoder, multi-dimensional node features are constructed; at the same time, based on the relationship type prediction, confidence estimation and subject-object spatial relative difference, edge features are generated, providing standardized input containing semantic, position and relationship information for subsequent graph structure modeling;

[0035] The generation of entity representation is represented as:

[0036]

[0037] The generation of relationship representation is:

[0038]

[0039] wherein, represents the abstract semantic features output by the last layer of the decoder, d model is the feature dimension, is the normalized bounding box coordinate, including the center coordinate and the width and height, is the object class probability distribution, C is the number of classes; in the edge feature Indexing for discrete relation types, Indexing for relation confidence, and respectively, the coordinate difference between the subject and object and the aspect ratio, after splicing, constitute the relative characteristics of the 4-dimensional space.

[0040] Further, the step S3 comprises:

[0041] S301: Feature screening

[0042] By the node confidence threshold τ node and the edge confidence threshold τ edge , high-confidence entities and relations are screened out; entities with node category probability higher than the threshold are retained, low-confidence edges are removed, noise introduced by false positives or ambiguous relations is reduced, ensuring that the features input into the graph convolutional network have high reliability, providing a data basis for subsequent semantic fusion;

[0043] S302: Feature normalization

[0044] The bounding box coordinates are scaled to the [0, 1] interval according to the image size B, eliminating the influence of image resolution differences on spatial positioning; the subject and object aspect ratios are logarithmically processed to alleviate the numerical instability problem caused by extreme size differences; the normalization operation unifies the feature dimension, enhances the sensitivity of the model to spatial features, and accelerates the training convergence process, improving the stability of the classification task.

[0045] Further, the step S4 comprises:

[0046] S401: Input of graph neural network model

[0047] The input of the graph neural network is the screened node feature Node filtered and edge feature Edge filtered ;

[0048] S402: Message passing and feature aggregation

[0049] The node features are updated through the message passing mechanism, each node aggregates the information of its neighbor nodes, and the feature is updated through the graph neural network;

[0050] S403: Generation of graph-level features

[0051] The node features are aggregated using the global average pooling method to obtain the global representation of the graph

[0052] S404: Modeling of binary classification task

[0053] The graph-level features are input into the fully connected layer, and the Sigmoid function is mapped to the sentiment binary classification probability wherein The closer to 1 indicates the higher probability of positive sample of the image, and the closer to 0 indicates the lower probability of positive sample, and the classification result directly reflects the overall sentiment tendency of the street view image sample;

[0054] S405: Loss function and training

[0055] The model is trained separately, that is, the model is independently trained in different sentiment dimensions (such as six sentiment dimensions of the PP2 data set), and a binary cross-entropy loss function is used to optimize the model, wherein y i ∈{0,1} is the real sentiment label in the corresponding dimension; the weights and classification parameters of the end-to-end adjustment graph convolution network layer are completed by back propagation, the interference of low confidence features on classification is suppressed, and the perception ability of the model to complex street view semantics is enhanced, in the training process, the mapping relationship between the graph structure features and the sentiment labels is gradually strengthened, and finally the high-precision classification of the street view image in each sentiment dimension is realized.

[0056] A sentiment perception quantization system based on graph structure considering scene semantic information, the system is used to execute any of the above methods, the system comprises:

[0057] The sentiment perception street view image data set preprocessing module: using an existing street view image sentiment perception data set, the data set needs to contain each image and its corresponding sentiment score label, the continuous sentiment score label is converted into a binary classification label, and an image data set with positive and negative sentiment labels is obtained;

[0058] The high-dimensional feature extraction module: for the street view image in the existing sentiment perception data set, the general visual features are extracted by using the backbone network and the position coding is performed, then the pre-training weights of the trained scene graph generation model are used to extract the high-level semantic features of the image, and the multi-layer feature map, the migrated high-dimensional features, the position coding and the embedding representation of the entity and the relationship are obtained;

[0059] The scene graph generation module: based on the output of the high-dimensional feature extraction module, the node features and edge features required by the graph convolution network are generated by extracting and fusing the semantic features of the entity and the relationship information between the entities, that is, the generated entity representation and relationship representation;

[0060] The street view entity prediction module: the generated entity features and relationship features are filtered and standardized, a high-quality graph structure is constructed, and the features input into the graph convolution network have high confidence and consistency, so that the node features and edge features with high confidence and the filtered graph structure are obtained;

[0061] Street view entity relationship prediction module: through the screened graph structure, high confidence node features and edge features are used for information transmission and aggregation to generate a global representation of the graph, finally, the generated global representation is input into a fully connected layer, and a graph dataset with positive and negative emotion labels is used for training, and a sigmoid function is used for binary classification prediction of emotion to realize efficient perception and quantification of street view image sentiment.

[0062] A computer device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of the preceding aspects when executing the program.

[0063] A computer readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method of any one of the preceding aspects.

[0064] Compared with the prior art, the advantages of the present application are that:

[0065] Firstly, the street view dataset with positive and negative emotion labels is used to ensure that the model obtains real and reliable emotion standards during training, thereby improving the accuracy and reliability of prediction.

[0066] Secondly, the scheme uses a pre-trained backbone network and a scene graph generation model to extract multi-level visual features and high-level semantic information. The deep feature learning capability enables the model to better capture the emotional features in complex street view images.

[0067] Through the graph convolution network, the scheme effectively models the relationship between nodes, fully excavates the semantic and spatial information of entities, thereby enhancing the effect of emotion classification. At the same time, the feature screening and normalization processing improve the confidence of the input features, significantly reduce the influence of noise, and enhance the robustness of the model.

[0068] The flexibility and scalability of the scheme enable it to adapt to analysis of different emotion dimensions, facilitating subsequent secondary development. In addition, the explicit feature extraction and graph construction process enhances the explainability of the model, enabling users to understand the basis of emotion prediction and improving user trust.

[0069] In summary, the present application constructs an efficient, accurate and well-explainable emotion perception model suitable for various application scenarios of modern intelligent cities and transportation systems. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The main flowchart of the emotion perception quantification method based on graph structure considering scene semantic information of the present application. Detailed Implementation

[0071] The specific implementation of the present invention is described below with reference to embodiments:

[0072] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0073] Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity of description and are not intended to limit the scope of the invention. Any changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0074] Example 1:

[0075] like Figure 1 As shown, the present invention provides a method and system for quantifying emotion perception that takes into account scene semantic information based on graph structure, including the following steps:

[0076] S1. Prepare a dataset (street view format) that can represent the emotional perception status of a certain area, as the data source for training the emotional perception model. Then, process the score labels of the dataset and convert them into positive and negative sample labels for binary classification tasks.

[0077] S2. Using the dataset in S1 as input data, extract general visual features and perform position encoding using the backbone network. Use the trained scene graph generation model to extract high-level features of each image in the dataset and generate entity representations and relationships between entities.

[0078] S3. Select the intermediate features and prediction results from the previous step to generate node features and edge features, and fuse multi-dimensional features to construct node features and edge features.

[0079] S4. Set filtering rules to filter node features and edge features to ensure that the subsequent graph convolutional neural network receives high-quality training features.

[0080] S5. Using the graph structure information generated in S4 as input, a graph convolutional neural network is used to fuse the visual semantic information of the image to perform the classification task of street scene emotion perception images.

[0081] The specific label conversion operations in S1 are as follows:

[0082] For the existing street view images and labels in the emotional perception dataset, a certain image score is marked as S i , L is the lower bound of positive samples, U is the upper bound of negative samples, μ score and σ score are the mean and standard deviation of all image scores in a certain emotional dimension, and y i is the label of the converted image i.

[0083]

[0084] K=μ score -δ·σ score ,

[0085] U=μ score +δ·σ score

[0086]

[0087] wherein the specific description of S2 is as follows:

[0088] Selecting a backbone network

[0089] In order to capture the multi-scale visual information of the image, ResNet50 is selected as the backbone network. ResNet50 can effectively alleviate the gradient vanishing problem in deep network by virtue of its residual connection structure, and gradually extract hierarchical features from local details to global semantics through stacking convolution layers.

[0090] The image is input into the backbone network (backbone), and Resnet50 is used to extract multi-layer feature maps of the image. The extracted image features are wherein H and W are the height and width of the image respectively, and C is the number of channels.

[0091] The deep features F i of ResNet50 contain rich semantic information (such as object categories, textures) and spatial information (such as contour positions), providing a high-quality visual feature basis for subsequent Transformers.

[0092] Transfer learning

[0093] The core of transfer learning is to use the general knowledge of the pre-trained model to accelerate the convergence of the target task and improve the model generalization ability. In the system and method explained in the present invention, two steps involve transfer learning.

[0094] (1) Visual feature transfer: the general visual features learned by the ResNet50 pre-trained model on the ImageNet dataset are transferred to the target task, so that the model can more accurately extract high-dimensional features in the image and perform target recognition. The formulaic representation of transfer learning is:

[0095]

[0096] where denotes the image feature transferred from the ImageNet dataset, denotes the image feature in the target task data after transfer.

[0097] (2) Semantic knowledge transfer: The general scene semantic information in the VisualGenome (VG) dataset and the prior information about street view are transferred to the scene graph generation model, so that the model can more accurately express the objects and their mutual relationships in the street view image. The formalized representation of transfer learning is:

[0098]

[0099] where denotes the entity feature representation transferred from the VG dataset, denotes the entity feature representation in the street view data after transfer.

[0100] Position encoding

[0101] Convolutional neural networks lack the perception of the location of pixel information in the image, while the target detection task is highly dependent on spatial information. Position encoding helps the model to distinguish objects at different positions and model spatial relationships by explicitly injecting coordinate information.

[0102] Sine encoding and learning-based position encoding are used to generate position encoding of image pixel positions, converting pixel images into high-dimensional vectors containing spatial information of the image

[0103] where, for each position x and y, the sine and cosine functions are used for encoding respectively:

[0104]

[0105] where, x embed and y embed are the spatial coordinates of the image in the horizontal and vertical directions respectively, and dim t is the encoding scale generated in the feature dimension.

[0106] Similarly, learning-based position encoding can also be used to generate position encoding by embedding each position of the image into a trained vector space.

[0107] P = rowembed(i) + colembed(j)

[0108] where i is the column coordinate, j is the row coordinate, rowembed(i) and colembed(j) represent the row and column embedding vectors, respectively. The generated spatial position encoding with the image feature F i In combination, spatial information is provided for each image position.

[0109] Further, the spatial position encoding incorporates the feature position-aware information generated by the image feature F i + P i After adjusting the number of channels to adapt to the input dimension of the Transformer, this feature is input into the encoder of the Transformer, which processes the feature position-aware information through a self-attention mechanism, learns the relationship between different positions in the image, converts the local features extracted by the Backbone into feature representations with global context awareness, fuses semantic and spatial information, and provides a rich context basis for the decoder. The calculation process of the encoder can be represented as:

[0110] Q, K, V = Linear(inputfeatures)

[0111]

[0112] where Q, K, V are the query, key and value, respectively, which are obtained from the input feature Inputfeatures through linear transformation. In this way, the Transformer encoder can capture the long-range dependencies between image features.

[0113] After processing by the decoder, these high-dimensional feature representations are decoded into more specific embedding representations. Specifically, the input of the decoder includes not only the output src (local abstract features) from the encoder, but also the position encoding P target generated earlier, which is used to tell the decoder the specific position of the target in space.

[0114] The input target embedding X target in the decoder, the self-attention layer calculates the self-attention mechanism:

[0115] Q target ,K target ,V target = Linear(X target )

[0116] The output is:

[0117]

[0118] Next, a cross-attention layer is used, taking the image features X from the encoder as input. image and target embedding X target Calculate cross-attention:

[0119] Q target ,K image V image =Linear(X) target ),Linear(X image )

[0120] The output is:

[0121]

[0122] Feedforward Neural Network (FFN):

[0123] Apply a feedforward neural network (FFN) transformation to the position of each target:

[0124] FFN(X target ) = max(0,X target W1+b1)W2+b2

[0125] Layer normalization is applied to the outputs of self-attention and cross-attention:

[0126]

[0127] After processing by the decoder, the multi-layer output hs (HiddenStates) of the entity decoder and the multi-layer output hs of the triplet decoder are obtained. t (Hidden States triplets) predict entity categories through a linear layer, and a multilayer perceptron generates the coordinates of a four-dimensional bounding box, which are then normalized to [0,1] using a sigmoid function; the entity decoder outputs... Multi-layer decoding results are used to predict object categories and bounding boxes. Triple decoder output. After splitting, independent representations of the subject and object are obtained, which are used for relation prediction.

[0128] Furthermore, suitable features generated by the Transformer and the prediction results of the prediction head are selected as the source of feature construction for subsequent node features and edge features, providing high-quality training data input for subsequent graph convolutional neural networks.

[0129] S3 is explained in detail below:

[0130] S31 Node Feature Generation

[0131] The node feature is used to represent the detected entity (object) and is composed of the following three parts, forming a high semantic + spatial perception node feature:

[0132] (1) The output hs of the last layer of the decoder decoder is taken as the abstract feature nodes hs of the entity, which is represented as:

[0133]

[0134] Where B is the batch size, N q is the preset query number, d model is the feature dimension, hs[-1] is the output of the last layer of the decoder, which contains the abstract semantic information of the last layer.

[0135] (2) The bounding box coordinates: using the bounding box regression head, the normalized coordinates of the bounding box nodeboxes are generated after Sigmoid normalization, providing the position feature information of the entity, which is represented as:

[0136]

[0137] Where σ is the Sigmoid function, and the normalized bounding box coordinates [c x ,c y ,w, h] are output, is the weight matrix of the bounding box regression head.

[0138] (3) Class probability: using the class probability regression head, the object class probability nodeclass probs is obtained after Softmax processing, which is represented as:

[0139]

[0140] Where C is the number of object categories, excluding the background class, where is the weight matrix of the classification head.

[0141] After concatenating the above features, the information of "what is it" (category), "where is it" (coordinates) and "how to associate" is encoded, providing multi-dimensional information for the graph convolution network to obtain the final node feature matrix, which can be represented as follows:

[0142]

[0143] S32 edge feature generation

[0144] The edge feature is used to describe the relationship between entities, which needs to depict the relationship type, confidence and spatial interaction mode between entities, and support the graph network to reason about complex relationships, which is composed of the following three parts:

[0145] (1) Relationship type: the prediction result using the relationship classification head, denoted as:

[0146]

[0147] where N r is the number of triples, each value is the predicted relationship class index, is the original output of the relationship classification head, R is the number of relationship classes

[0148] (2) Relationship confidence:

[0149] denoted as:

[0150]

[0151] (3) Spatial relative feature (referring to the difference and difference of the subject and object in space, used to represent the relationship feature difference between the subject and object):

[0152] Based on the bounding box of the subject and object, it can be represented as:

[0153]

[0154] The above features are spliced into edge features, denoted as:

[0155]

[0156] Further, S4 is specifically described as follows:

[0157] S41 Feature selection:

[0158] (1) Node feature selection:

[0159] Keep the entity features with high confidence, set the confidence threshold nodeτ, and the selection process of the node feature can be represented as:

[0160] validnoses=nodefeatures[max(node class probs,dim=-1)>τ node ]

[0161] (2) Edge feature selection:

[0162] Keep the relationship features with high confidence, set the confidence threshold edgeτ, and the selection process of the edge feature can be represented as:

[0163] valid edges=edge features[edge conf>τ edge ]

[0164] S42 Feature normalization:

[0165] Screening removes noise detection, reduces the computational redundancy of graph convolution network; normalization ensures the balance of different feature scales, accelerates model convergence.

[0166] (1) Coordinate normalization

[0167] (S is the input image size)

[0168] (2) Logarithmic aspect ratio

[0169] wh ratio = log(wh ratio + 1)

[0170] Further, the specific description of S5 is as follows:

[0171] S51: Based on the node features and edge features screened in S4, the graph structure information is constructed. Define the graph G = (V, E), where:

[0172] Node set V: Each node corresponds to an effective entity, and the node feature is the screened where N v is the number of effective nodes, d model is the dimension of the abstract feature output by the decoder, 4 is the normalized coordinate, and C is the number of object categories.

[0173] Edge set E: Each edge corresponds to an effective relationship screened, and the edge feature is where N e is the number of effective edges, and the 6-dimensional feature includes relationship type, confidence and spatial relative feature (Δ xy , wh ratio ).

[0174] Adjacency matrix Based on the edge feature construction, if there is an edge between nodes i and j, then A ij = edge conf x onehot(edgetype), that is, the joint weight represented by the relationship confidence and type coding represents the connection strength.

[0175] S52 message passing and feature aggregation

[0176] Through multi-layer graph convolution operation, the neighborhood information is aggregated, the node features and edge features are fused, and the high-order graph representation is generated. The specific steps are as follows:

[0177] Message passing:

[0178] When the graph convolution neural network updates the node features, the formula can be used to explain:

[0179]

[0180] Among them:

[0181] I is the identity matrix. is the degree matrix of the added self-loop adjacency matrix, is the node feature of the l-th layer, the initial input H (0) = nodefeatures. is a trainable weight matrix. σ is an activation function.

[0182] Multi-layer feature stacking:

[0183] Stacking LL layers of GCN to capture multi-hop neighborhood information, and the final output node feature is:

[0184]

[0185] S53 graph-level feature generation

[0186]

[0187] S54 binary classification task modeling

[0188] Input the graph-level feature into the fully connected layer and the Sigmoid function, and output the sentiment perception classification probability

[0189]

[0190] b cls ∈R, is the classification layer parameter, represents the probability that the image belongs to the "positive emotion" category, represents the probability that the image belongs to the "positive emotion" category.

[0191] S55 loss function and training

[0192] The binary cross-entropy loss function is used to optimize the model:

[0193]

[0194] where y i ∈{0,1} is the true label, and the model updates the GCN weight {W (l)}, the classification layer parameter W cls , b cls through backpropagation, achieving end-to-end training.

[0195] Example 2:

[0196] ​The application provides a sentiment perception quantification system based on a graph structure and considering scene semantic information, which can be used to implement the sentiment perception quantification method based on a graph structure and considering scene semantic information.

[0197] The emotion perception street view image dataset preprocessing module: based on the street view images and corresponding score labels in the original dataset, the original score labels of the street view images are converted into binary classification labels by filtering noise data and setting a threshold.

[0198] The high-dimensional feature extraction module: extracts general high-dimensional visual features of the street view images.

[0199] The scene graph generation module includes the following sub-modules:

[0200] The module takes the street view images in the dataset as input, processes the input street view images using the high-dimensional feature extraction module, and predicts the scene entities and the relationships between the entities by using the generated high-dimensional visual features combined with the attention mechanism.

[0201] The street view entity prediction module: uses the attention mechanism combined with the encoder and the entity decoder to predict the street view entities.

[0202] The street view entity relationship prediction module: uses the attention mechanism combined with the encoder and the triple decoder to predict the relationships between the street view entities.

[0203] The graph-level feature integration generation module: combines the intermediate abstract semantic features and the prediction results of the scene graph generation module to generate graph-level features for training the sentiment perception graph convolutional network.

[0204] The graph convolutional emotion semantic learning module:

[0205] The graph convolutional emotion semantic learning module:

[0206] Embodiment 3:

[0207] The embodiment provides a terminal device, which comprises a processor and a memory for storing a computer program, the computer program comprising program instructions, and the processor is used for executing the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are particularly suitable for loading and executing one or more instructions to implement a corresponding method flow or a corresponding function; the processor in the embodiment of the application can be used for the operation of the emotion perception quantification method based on the graph structure considering the scene semantic information, comprising the following steps:

[0208] S1: for the existing street view image in the emotion perception data set, the general visual features are extracted by using the backbone network and the position coding is performed, then the pre-training weight of the trained scene graph generation model is used to extract the high-level semantic features of the image, and the multi-layer feature map, the high-dimensional feature after migration, the position coding and the embedding representation of the entity and the relationship are obtained;

[0209] S2: based on the output of step S1, the node features and edge features required by the graph convolution network are generated by extracting and fusing the semantic features of the entity and the relationship information between the entities, that is, the generated entity representation and relationship representation;

[0210] S3: the generated entity features and relationship features are screened and standardized to construct a high-quality graph structure, so that the features input into the graph convolution network have high confidence and consistency, thereby obtaining high-confidence node features and edge features, and a screened graph structure;

[0211] S4: by using the high-confidence node features and edge features, information transmission and aggregation are performed through the screened graph structure to generate a global representation of the graph, finally, the generated global representation is input into a fully connected layer, and the image data set with positive and negative emotion labels is used for training, and a sigmoid function is used for emotion binary classification prediction, so as to realize efficient perception and quantification of the street view image emotion tendency.

[0212] Embodiment 4:

[0213] The embodiment provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a terminal device, used for storing programs and data. It can be understood that the computer readable storage medium herein can include a built-in storage medium in the terminal device, and of course can also include an expansion storage medium supported by the terminal device. The computer readable storage medium provides a storage space, which stores an operating system of the terminal. Furthermore, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory or a non-volatile memory, for example, at least one disk memory.

[0214] The one or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to realize the corresponding steps of the emotion perception quantization method based on the graph structure considering scene semantic information in the above embodiment; the one or more instructions in the computer readable storage medium are loaded and executed by the processor to perform the following steps:

[0215] S1: For the existing street view image in the emotion perception data set, general visual features are extracted by using a backbone network and position coding is performed, then the pre-training weight of the trained scene graph generation model is used to extract high-level semantic features of the image, to obtain a multi-layer feature map, a high-dimensional feature after migration, position coding and embedded representation of entities and relationships;

[0216] S2: Based on the output of step S1, node features and edge features required by a graph convolution network are generated by extracting and fusing semantic features of entities and relationship information between entities, that is, generated entity representation and relationship representation;

[0217] S3: The generated entity features and relationship features are screened and normalized to construct a high-quality graph structure, so that the features input into the graph convolution network have high confidence and consistency, thereby obtaining high-confidence node features and edge features and a screened graph structure;

[0218] S4: Through the screened graph structure, information transmission and aggregation are performed by using the high-confidence node features and edge features to generate a global representation of the graph, finally, the generated global representation is input into a fully connected layer, and an image data set with positive and negative emotion labels is used for training, and a Sigmoid function is used for emotion binary classification prediction, so as to realize efficient perception and quantization of the street view image emotion tendency.

[0219] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In one embodiment, the present application can be implemented in software and can be stored on a computer readable medium, which can include random access memory (RAM), read only memory (ROM), magnetic disk or optical disk, or the like. The software implementation of the present application files can further be transmitted or received over a modem or network connection.

[0220] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0221] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0222] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0223] While the application has been described in detail with respect to the preferred embodiments thereof, it will be apparent to those skilled in the art that various alterations, modifications, and improvements can be made therein without departing from the spirit and scope of the application.

[0224] Many other changes and modifications can be made to the application without departing from the spirit and scope of the application. It is therefore intended to cover in the appended claims all such changes and modifications that are within the scope of this application.

Claims

1. A graph-based method for quantifying sentiment perception that takes into account scene semantic information, characterized in that, The method includes: S1: For street scene images in the existing emotion perception dataset, extract general visual features and perform position encoding using a backbone network. Then, use the pre-trained weights of the trained scene graph generation model to extract high-level semantic features of the image, and obtain multi-layer feature maps, transferred high-dimensional features, position encoding, and embedding representations of entities and relationships. S2: Based on the output of step S1, by extracting and fusing the semantic features of entities and the relational information between entities, the node features and edge features required by the graph convolutional network are generated, that is, the generated entity representation and relational representation. S3: Filter and normalize the generated entity features and relation features to construct a high-quality graph structure, ensuring that the features input to the graph convolutional network have high confidence and consistency, thereby obtaining high-confidence node features and edge features, as well as the filtered graph structure; S4: Using the filtered graph structure, information is transmitted and aggregated using high-confidence node features and edge features to generate a global representation of the graph. Finally, the generated global representation is input into a fully connected layer and trained using an image dataset with positive and negative sentiment labels. The Sigmoid function is used to perform binary classification prediction of sentiment to achieve efficient perception and quantification of sentiment tendencies in street view images.

2. The sentiment perception quantification method based on graph structure that takes into account scene semantic information as described in claim 1, characterized in that, Before step S1, an existing street view image sentiment perception dataset is used. The dataset contains each image and its corresponding sentiment score label. The continuous sentiment score labels are converted into binary labels to obtain an image dataset with positive and negative sentiment labels.

3. The emotion perception quantification method based on graph structure that takes into account scene semantic information as described in claim 2, characterized in that, The specific tag conversion process is as follows: Given street view images and labels in an existing emotion perception dataset, a certain image score is labeled S. i L is the lower bound for positive samples, U is the upper bound for negative samples, and μ score and σ score Let y be the mean and standard deviation of all image scores under a certain emotional dimension. i The label for the converted image i. L=μ score -d·s score , U=μ score +d·s score 4. The sentiment perception quantification method based on graph structure that takes into account scene semantic information according to claim 1, characterized in that, Step S1 includes: S101: Feature Extraction The image is input into the backbone network, and multi-layer feature maps of the image are extracted. The extracted features are: Where H and W are the height and width of the image, respectively, and C is the number of channels; S102: Transfer Learning The general visual features learned on the ImageNet dataset by the pre-trained model using ResNet as the backbone network are transferred to the target task, enabling the model to extract high-dimensional features from images and perform target recognition more accurately; represented as: By transferring general scene semantic information and prior information about street scenes from the Visual Genome (VG) dataset to the scene graph generation model, the model can more accurately represent objects and their relationships in street scene images; represented as: X VG →X SS Where X VG X represents general scene semantic information transferred from the VG dataset. SS This represents the semantic representation of the scene in the migrated street view data; S103: Location Code Generate spatial location codes for the extracted feature maps Used to capture spatial information in an image, p is the dimension of the position encoding, which is generated by sinusoidal position encoding; S104: Generate embedded representation The extracted feature map F i and spatial location coding P i The input is fed into a Transformer-based scene graph generation model to generate an embedding representation X for each entity and relation. entities and R relations .

5. The sentiment perception quantification method based on graph structure that takes into account scene semantic information according to claim 1, characterized in that, Step S2 includes: Multi-dimensional features are extracted and fused from the output of the scene graph generation model to generate the node and edge features required by the graph convolutional network, providing semantic, spatial and relational information for subsequent graph structure modeling; By integrating the abstract semantic features output by the decoder, the bounding box coordinate regression results, and the object category probability, multidimensional node features are constructed; at the same time, edge features are generated based on relation type prediction, confidence estimation, and the relative differences between the subject and object spaces, providing a standardized input containing semantic, positional, and relational information for subsequent graph structure modeling. The entity representation is generated as follows: Generation of relational representations: in, d represents the abstract semantic features output by the last layer of the decoder. model For feature dimension, The bounding box coordinates are normalized, including center coordinates and width and height. Let C be the probability distribution of object categories, and C be the number of categories; in the edge features For discrete relation type indexes, c∈[0,1] Nr For relationship confidence, and These are the center coordinate difference between the subject and object and their aspect ratio, which, when spliced ​​together, form a 4-dimensional spatial relative feature.

6. The sentiment perception quantification method based on graph structure that takes into account scene semantic information according to claim 1, characterized in that, Step S3 includes: S301: Feature Filtering Using the node confidence threshold τ node And edge confidence threshold τ edge The system filters out entities and relationships with high confidence; retains entities whose node category probability is higher than the threshold, removes low-confidence edges, reduces noise introduced by false detections or ambiguous relationships, and ensures that the features of the input graph convolutional network have high reliability, providing a data foundation for subsequent semantic fusion. S302: Feature Normalization The bounding box coordinates are scaled to the [0,1] range according to the image size B to eliminate the influence of image resolution differences on spatial localization; the aspect ratio of the subject and object is logarithmically processed to alleviate the numerical instability caused by extreme size differences; the normalization operation unifies the feature units, enhances the model's sensitivity to spatial features, accelerates the training convergence process, and improves the stability of the classification task.

7. The sentiment perception quantification method based on graph structure that takes into account scene semantic information according to claim 1, characterized in that, Step S4 includes: S401: Input to the graph neural network model The input to the graph neural network is the filtered node features. filtered and edge features filtered ; S402: Message Passing and Feature Aggregation Node features are updated through a message passing mechanism. Each node aggregates information from its neighboring nodes and updates its features through a graph neural network. S403: Graph-level Feature Generation Global average pooling is used to aggregate node features to obtain a global representation of the graph. S404: Binary Classification Task Modeling Graph-level features are input into a fully connected layer and mapped to sentiment binary classification probabilities using a Sigmoid function. in The closer to 1, the higher the probability of a positive sample in the image; the closer to 0, the lower the probability of a positive sample. The classification result directly reflects the overall sentiment of the street view image samples. S405: Loss Function and Training By training the models separately for different sentiment dimensions, and optimizing the models using a binary cross-entropy loss function, y i ∈{0,1} represents the true sentiment label in the corresponding dimension; the weights and classification parameters of the graph convolutional network layer are adjusted end-to-end through backpropagation to suppress the interference of low-confidence features on classification, enhance the model's ability to perceive complex street scene semantics, and gradually strengthen the mapping relationship between graph structure features and sentiment labels during training, ultimately achieving high-precision classification of street scene images in various sentiment dimensions.

8. A graph-based emotion perception quantification system that takes into account scene semantic information, characterized in that, The system is used to perform the method according to any one of claims 1-7, the system comprising: The preprocessing module for the emotion-aware street view image dataset uses an existing street view image emotion-aware dataset. The dataset contains each image and its corresponding emotion score label. The continuous emotion score labels are converted into binary labels to obtain an image dataset with positive and negative emotion labels. High-dimensional feature extraction module: For street scene images in the existing sentiment perception dataset, the backbone network is used to extract general visual features and perform position encoding. Then, the pre-trained weights of the trained scene graph generation model are used to extract high-level semantic features of the image, resulting in multi-layer feature maps, transferred high-dimensional features, position encoding, and embedding representations of entities and relationships. Scene graph generation module: Based on the output of the high-dimensional feature extraction module, it extracts and fuses the semantic features of entities and the relational information between entities to generate the node features and edge features required by the graph convolutional network, that is, the generated entity representation and relational representation; Street view entity prediction module: It filters and normalizes the generated entity features and relationship features, constructs a high-quality graph structure, and ensures that the features input to the graph convolutional network have high confidence and consistency, thereby obtaining high-confidence node features and edge features, as well as the filtered graph structure; Street view entity relationship prediction module: Through the filtered graph structure, information is transmitted and aggregated using high-confidence node features and edge features to generate a global representation of the graph. Finally, the generated global representation is input into a fully connected layer and trained using an image dataset with positive and negative sentiment labels. The Sigmoid function is used to perform binary classification prediction of sentiment to achieve efficient perception and quantification of sentiment tendencies in street view images.

9. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the emotion perception quantification method based on graph structure that takes into account scene semantic information, as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the emotion perception quantification method based on graph structure and considering scene semantic information, as described in any one of claims 1 to 7.

Images