A subjective review value editing method driven by value system
By constructing a multimodal value view scenario diagram and a self-coding adversarial network, combining value conditions and emotional information, the problem of insufficient fusion and controllability of multimodal information in the existing comment generation technology is solved, and the generation of subjective emotions is achieved and comments on value tendencies.
Patent Information
- Application Number
- CN202411161518.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-08-23
AI Technical Summary
Existing comment generation technology is difficult to effectively integrate multimodal information such as vision and text. The generated comments are low in generalization and poor controllability, and fail to make full use of the subjective information and value system views in the content.
A subjective comment generation method driven by value system is proposed. By constructing a multimodal value view scenario map, introducing value condition constraints and emotional information, designing a self-coding adversarial network, realizing the end-to-end from multimodal information to comment text Map and optimize the generation of results using comment sentiment discriminator and value evaluator.
The generated comments are rich in subjective emotions and value tendencies, and the emotional tone is controllable, which meets the user's emotional expression needs. They can more comprehensively capture the subjective attributes of multimedia content and improve the readability and value orientation of comments.
Smart Images

Figure CN118690755B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of comment generation, and in particular to a subjective comment value editing method driven by a value system. Background Art
[0002] Commenting on social media platforms has become an important way for people to express their opinions and share their feelings, but social media platforms lack corresponding methods to guide comments.
[0003] Comment generation refers to the task of automatically generating relevant comments based on multimedia content using artificial intelligence technology. Its goal is to generate comment text that is relevant to the content, semantically coherent, emotionally appropriate, and subjectively meaningful. At present, research on comment generation mainly focuses on objectively describing the content itself, and rarely considers using subjective information such as implicit emotional attitudes, value systems, and viewpoints to guide comments. In addition, most existing methods use a sequence-to-sequence encoding and decoding framework, which makes it difficult to effectively integrate multimodal information such as vision and text, and the generated comments often have low generalization and poor controllability. Summary of the invention
[0004] In order to solve the above technical problems, the present invention provides a method for editing the value of subjective comments driven by a value system. First, the method mines the emotional tendencies and viewpoints contained in the multimodal information of the video, such as vision and text, and constructs a unified value viewpoint scene graph representation. On this basis, the value system-related vocabulary is introduced as a generation constraint, and an end-to-end mapping from the self-encoding adversarial network learning scene graph to the comment text is designed. At the same time, the comment sentiment discriminator is used to evaluate the emotional tendency and value orientation of the generated comments, and guide the comment generation model to generate positive subjective comments that are in line with the value system. The present invention can provide a new technical means for purifying the video comment ecology.
[0005] The key points of the present invention are:
[0006] 1. Propose a video comment self-encoding generation network that incorporates value condition constraints. By introducing positive vocabulary conditions, it guides the generation of subjective comment texts that are rich in subjective emotions and conform to the value system;
[0007] 2. Introduce a conditional adversarial learning mechanism that integrates emotions and value systems, and improve the subjective expression ability and value guidance of comments through the game between the generator and the discriminator;
[0008] 3. Build an end-to-end video-to-comment generation paradigm to achieve controllable transformation mapping from multimodal value perspective scene graph to comment text.
[0009] The present invention specifically adopts the following technical solutions:
[0010] A subjective comment value editing method based on a value system drive, inputting a given video and preset value conditions into a trained comment generation model, and outputting a subjective comment text that meets the value conditions; the training process of the comment generation model includes the following steps:
[0011] Step 1: Extract multimodal information of video V and obtain visual scene graph through visual modality sentiment analysis , through text modal value point of view mining to obtain text point of view graph , through the visual scene graph , Text Viewpoint Diagram And combined with the external value system knowledge graph , build a value perspective scene graph ;
[0012] Step 2: View the scene graph from a value perspective As input, a self-encoding generation network consisting of an encoder and a Transformer decoder is used to generate subjective review text that matches the value conditions, including:
[0013] Value condition embedding: Map the value condition to Dimensional value condition embedding ;
[0014] Scene graph encoding with emotional information: The encoder uses a graph neural network based on Value point scene diagram Perform embedded representation learning to obtain the encoding result H that incorporates emotional information;
[0015] Decoding stage value condition interaction: embed the encoding result H and value condition output by the encoder As the input of the Transformer decoder, the Transformer decoder generates a comment word at each moment in an autoregressive manner, and then obtains the final subjective comment text;
[0016] Loss function of the autoencoder generation network Use cross entropy loss with regularization:
[0017] ;
[0018] is the conditional probability of the comment word output by the Transformer decoder, for Comments generated at all times, express The comment words generated before time, Indicates time 0 Comments generated at all times, is the comment sentiment discriminator, Represents the conditional probability All , is the weight coefficient.
[0019] Furthermore, it also includes introducing an adversarial learning mechanism in the training process of the comment generation model, scoring the subjective comment text generated by the autoencoder generation network through the comment sentiment discriminator, and optimizing the autoencoder generation network through the scoring results, specifically including:
[0020] Let y be the subjective review text generated by the self-encoding network, and the corresponding value condition embedding is , is the real review text sampled from the real review dataset, and the corresponding value condition embedding is , the comment sentiment discriminator D is embedded with subjective comment text y and value condition For input, or with real comment text and value condition embedding As input, calculate the similarity between the generated subjective comment text or the real comment text and the value condition embedding, determine the source of the comment text and evaluate the emotional tendency:
[0021] ; ;
[0022] in, is the sigmoid activation function, Scoring function for comment sentiment discriminator; objective function for adversarial learning training of comment generation model for:
[0023] ;
[0024] That is, the autoencoder generation network G attempts to generate subjective review texts close to real review texts to deceive the review sentiment discriminator, while the review sentiment discriminator D attempts to distinguish between real review texts and subjective review texts, where and They represent that the self-encoder generation network G should minimize the objective function and the comment sentiment discriminator D should maximize the objective function; represents the expected score of the comment sentiment discriminator D for the real comment text r, represents sampling of real review text from the distribution of real review text dataset, Represents the distribution of real review text dataset; represents the expected score of the comment sentiment discriminator D on the generated subjective comment text y, represents the generated review text sampled from the probability distribution output by the autoencoder generation network, Represents a self-encoding network to value the scene graph and value condition embedding Conditional distribution for conditional prediction of subjective review text.
[0025] Furthermore, it also includes introducing a value evaluator Q based on the comment sentiment discriminator to calculate the value comment score, which is used to judge whether the subjective value orientation of the generated subjective comment text conforms to the value system, specifically including:
[0026] Let P be a series of descriptive sentences or keywords that reflect the value system. The value evaluator Q first calculates the real review text Similarity with P :
[0027] ;
[0028] in, It is the feature extraction component of the value evaluator; then a gating mechanism is added, when the similarity is below the threshold When the value estimator Q output is 0, Score the final value review:
[0029] ;
[0030] The objective function of adversarial learning training of the comment generation model is replaced by :
[0031] ;
[0032] It indicates that the value evaluator Q and the comment sentiment discriminator D need to maximize the objective function.
[0033] Furthermore, in step 1, extracting multimodal information of the video V specifically includes:
[0034] Using target detection technology and text extraction technology, we can obtain the visual object set in the video V and and a collection of text snippets , represents the nth visual object in the video V, represents the mth text segment in the video V; n and m represent the number of visual objects and the number of text segments in the video V, respectively.
[0035] Furthermore, in step 1, the visual scene graph obtained by visual modality sentiment analysis , specifically including:
[0036] The visual objects in the video V detected by the target detection model are used as object nodes in the visual scene graph. The image sentiment classification model is used to identify the value tendency of the visual objects. The obtained sentiment probability vector is embedded into the visual scene graph as the node attribute of the object node. :
[0037] ;
[0038] ;
[0039] in, is a collection of object nodes, Represents an object node and the sentiment probability vector Visual node tuple composed of A collection of Represents a connected object node The edge and spatial relationship representation of Visual edge tuples A collection of; For object nodes The emotion probability vector is the object node in the visual scene graph Node attributes of For the Object Nodes and Object Nodes The spatial relationship representation of is the edge attribute of the visual scene graph; .
[0040] Furthermore, in step 1, the text viewpoint graph is obtained by mining the text modal value viewpoints. , specifically including:
[0041] The text segment is used as a text node in the text opinion graph; the sentiment intensity of the text node is extracted using a sentiment intensity regression model; Text nodes The emotional intensity for:
[0042] ;
[0043] in, is the sentiment intensity regression model, ; At the same time, using the topic model Extracting text nodes The topic distribution :
[0044] ;
[0045] Then use the sentiment dictionary to calculate the sentiment correlation scores between text nodes and each topic:
[0046] ;
[0047] in, Represents a text node With theme The sentiment-related score of For text nodes Each word in Sentiment score; For a given topic distribution Time, word Belong to the theme The probability of the text node sentiment intensity and topic sentiment score vector are used as the node attributes of the text node in the text opinion graph to construct the text opinion graph :
[0048] ;
[0049] ;
[0050] in, is the kth text node The topic sentiment score vector contains all The sentiment score of each topic; is a collection of text nodes, Represents a text node , emotional intensity and the topic sentiment score vector A tuple of text nodes A collection of; For text nodes and text nodes The similarity of is the edge attribute of the text opinion graph; Represents a connected text node The edge and similarity Text edge tuples A collection of .
[0051] Furthermore, in step 1, the visual scene graph , Text Viewpoint Map Combined with the external value system knowledge graph , build a value perspective scene graph , specifically including:
[0052] Use the externally provided value system knowledge graph as the value system knowledge graph , value system knowledge graph Each concept node in represents a value system concept in the value system knowledge graph; the value tendency score of each concept node is calculated:
[0053] ;
[0054] Concept Node The value orientation score is the concept node in the value system knowledge graph. The node properties of For the concept node in the value system knowledge graph A collection of connected concept nodes, for The concept node in Concept Node The importance weight of Concept Node With concept nodes Then, the visual scene graph , Text Viewpoint Diagram and value system knowledge graph The node attributes and edge attributes in the value point scene graph are embedded into middle:
[0055] ;
[0056] ;
[0057] in, Value system knowledge graph The set of concept nodes in Respectively concept nodes and concept nodes, For object nodes With text nodes degree of relevance; Value system knowledge graph Medium Concept Node and concept nodes The relationship between is the value system knowledge graph Edge attributes of concept nodes , Concept Node and Composition of conceptual edge tuples ; A collection of tuples of nodes and node attributes representing the value point scene graph, including visual node tuples , text node tuple and a concept node tuple consisting of a concept node c and a value orientation score ; A set of tuples of edges and edge attributes representing the value point scene graph, including visual edge tuples , text edge tuple , by object node , text node and relevance The associated edge tuple , and the concept edge tuple .
[0058] Furthermore, in step 2, the encoder uses a graph neural network based on Value point scene diagram Perform embedded representation learning to obtain the encoding result H that incorporates emotional information, including:
[0059] The hidden states of the nodes in the value scene graph are updated and learned through a multi-layer graph neural network to obtain the encoding result H that integrates the emotional information. Tier The hidden state of a node The update formula is:
[0060] ;
[0061] in, Representation Node The set of indexes of neighbor nodes; Is a node Neighbor nodes The index of Indicates that in the l-1th layer, the node No. The hidden states of neighbor nodes; , For graph neural network The graph convolution parameter matrix of the layer is used to convert the hidden state of the nodes in the previous layer to the hidden state of the nodes in the current layer. For graph neural network The transformation matrix of the layer-value conditional embedding, is the activation function, Value point scene graph Midpoint and neighbor nodes adjacency matrix; by transforming the hidden states of neighboring nodes With the adjacency matrix and the graph convolution parameter matrix Multiply, aggregate the information of neighbor nodes into the hidden state update of the current node, and use the connection relationship between nodes to transfer and aggregate information;
[0062] After L layers of graph convolution operations of the graph neural network, the encoding result H is obtained.
[0063] Furthermore, the encoding result H output by the encoder and the value condition are embedded into As the input of the Transformer decoder, the Transformer decoder generates a comment word at each moment in an autoregressive manner, and then obtains the final subjective comment text, which includes:
[0064] set up for The hidden state of the Transformer decoder at time t; The calculation formula is:
[0065] ;
[0066] in, is a multi-head attention mechanism, , , Represent the query matrix, key matrix and value matrix in the attention mechanism respectively;
[0067] Conditional probability of review words output by Transformer decoder , represents the classification matrix, Represents the softmax function.
[0068] Compared with the prior art, the beneficial technical effects of the present invention are:
[0069] The present invention mines emotional tendencies and viewpoints from multiple modalities such as the visual and textual aspects of the video, and constructs a value viewpoint scene graph that integrates multiple subjective attributes. Compared with the analysis method of a single modality, the present invention can more comprehensively and accurately capture the emotional and viewpoint information contained in multimedia content, and provide a rich input representation for comment generation. Under the encoding-decoding framework, the present invention applies mechanisms such as value condition embedding and emotion-integrated scene graph encoding to the comment generation model to guide the generation of subjective comment texts consistent with the specified emotional tendencies. Through learnable value condition embedding and targeted decoding strategies, the present invention can flexibly control the emotional tone of the generated comments to better meet the user's emotional expression needs. In addition, the present invention uses the external value system knowledge graph to calculate the value tendency scores of multimedia content entities and themes, and integrates them into a unified value viewpoint scene graph representation. In the generation process, the present invention guides the selection and organization of themes, arguments, and evidence of the comment text in a targeted manner according to the user's personalized views or value orientations, so that the generated comments can reflect personalized emotional expression and viewpoint tendencies. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 Schematic diagram of the review generation model adopted by the present invention. DETAILED DESCRIPTION
[0071] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.
[0072] In view of the fact that the existing review generation methods fail to effectively utilize the subjective information contained in the content itself and the generated reviews are not controllable and value-oriented enough, this paper proposes a new subjective review value editing method based on value system drive. The purpose of this method is to:
[0073] (1) Fully explore the subjective emotions, opinions and positions contained in social media content, integrate external value system knowledge, and construct a unified value viewpoint-value scenario graph representation, laying the foundation for generating subjective comments rich in emotional attitudes and value system tendencies.
[0074] (2) Incorporate value system-related vocabulary into the comment generation process as constraints, and guide the model to generate positive comments that are in line with mainstream value orientations.
[0075] (3) Design an end-to-end autoencoder generation network to achieve controllable mapping from value scene graph to review text, so that the generated reviews are subjective, readable, and instructive.
[0076] This invention aims to promote content comment generation technology from objective description to subjective guidance, and to provide new technical means and application paths for purifying the social media ecology and spreading the value system.
[0077] like Figure 1 As shown, the overall technical solution of the present invention includes the following key steps:
[0078] 1. Multimodal value perspective mining and scene graph construction.
[0079] For the input video V, we first use target detection, text extraction and other technologies to obtain the visual object sets in it. and a collection of text snippets Multimodal information, represents the nth visual object in the video V, represents the mth text segment in the video V; n and m represent the number of visual objects and the number of text segments in the video V, respectively.
[0080] On this basis, we further combine the external value system knowledge graph Extract value orientation, explore the emotional tendencies, viewpoints and positions contained in each modal information, and construct a value perspective scene map The value perspective scene graph is a structured representation that integrates multimodal information and external value system knowledge. It describes the various elements in the visual scene and their relationships and value tendencies in the form of a graph. A collection of nodes representing a value perspective scene graph, including object nodes, text nodes, and concept nodes. The edge set representing the value view scene graph includes various semantic relationships between nodes.
[0081] 1.1, Visual modality sentiment analysis:
[0082] In the visual mode, the present invention uses the target detection model (here the YOLOv4 model can be used) to detect the visual objects in the video as object nodes in the visual scene graph, and uses the image sentiment classification model to identify the value tendency (positive, neutral) expressed by the visual objects. The sentiment probability vector is embedded in the visual scene graph as the attribute of the object node. :
[0083] ;
[0084] ;
[0085] in, is a collection of object nodes, Represents an object node and the sentiment probability vector Visual node tuple composed of A collection of Represents a connected object node The edge and spatial relationship representation of Visual edge tuples A collection of; For object nodes The emotion probability vector is the object node in the visual scene graph Node attributes of For the Object Nodes and Object Nodes The spatial relationship representation (such as up, down, left, right, etc.) is the edge attribute of the visual scene graph; .
[0086] In a preferred embodiment of the present invention, the image sentiment classification model may select a residual convolutional network (ResNet) or a visual Transformer (ViT).
[0087] 1.2, Text modal value perspective mining:
[0088] In the text mode, the present invention uses fine-grained sentiment analysis to extract the sentiment intensity of text fragments and uses topic models to mine their value perspectives. Text nodes The emotional intensity for:
[0089] ;
[0090] in, For sentiment intensity regression model, you can choose BERT-type natural language understanding model as sentiment intensity regression model. At the same time, use the topic model to extract the topic distribution of text nodes :
[0091] .
[0092] The topic model can use Latent Dirichlet Allocation (LDA). Then use the sentiment dictionary to calculate the sentiment correlation score between the text node and each topic:
[0093] ;
[0094] in, For text nodes Each word in Sentiment score; For a given topic distribution Time, word Belong to the theme The probability of the text node sentiment intensity and topic sentiment score vector are used as the node attributes of the text node in the text opinion graph to construct the text opinion graph :
[0095] ;
[0096] ;
[0097] in, is the kth text node The topic sentiment score vector contains all The sentiment score of each topic; is a collection of text nodes, Represents a text node , emotional intensity and the topic sentiment score vector A tuple of text nodes A collection of; For text nodes and text nodes The similarity of is the edge attribute of the text opinion graph; Represents a connected text node The edge and similarity A collection of text edge tuples.
[0098] 1.3, Multimodal value perspective scene graph fusion:
[0099] Finally, the visual scene graph , Text Viewpoint Diagram , Value system knowledge graph Fusion into a unified value view scenario graph .
[0100] Use the externally provided value system knowledge graph as the value system knowledge graph , value system knowledge graph Each concept node in the value system knowledge graph represents a value system concept; calculate the value tendency score of each concept node :
[0101] ;
[0102] Concept Node The value orientation score is the concept node in the value system knowledge graph. The node properties of For the concept node in the value system knowledge graph A collection of connected concept nodes, for The concept node in Concept Node The importance weight of Concept Node With concept nodes The relevance of value orientation.
[0103] Then, the visual scene graph , Text Viewpoint Diagram and value system knowledge graph The node attributes and edge attributes in the value point scene graph are embedded into middle:
[0104] ;
[0105] ;
[0106] in, Value system knowledge graph The set of concept nodes in Respectively concept nodes and concept nodes, For object nodes With text nodes degree of relevance; Value system knowledge graph Medium Concept Node and concept nodes The relationship between is the value system knowledge graph For example, for "food is placed on the table", the relationship between "food" and "table" is established. The value is binary code. A value of 0 indicates that there is no relationship between the two, and a value of 0 indicates that there is a relationship between the two. Concept node , Concept Node and Composition of conceptual edge tuples ; A collection of tuples of nodes and node attributes representing the value point scene graph, including visual node tuples , text node tuple and a concept node tuple consisting of a concept node c and a value orientation score ; A set of tuples of edges and edge attributes representing the value point scene graph, including visual edge tuples , text edge tuple , by object node , text node and relevance The associated edge tuple , and the concept edge tuple .
[0107] In the value perspective scene graph of the present invention, nodes represent different types of information elements such as visual objects, text fragments, and value system concepts. It integrates information from three modalities: vision, text, and knowledge, and can comprehensively depict the subjective emotions and value orientations conveyed by the video.
[0108] 2. Self-encoded comment generation based on sentiment conditions.
[0109] In this stage, multimodal value perspective scene graph As input, a subjective review text matching the specified value conditions is generated through an autoencoder generation network.
[0110] In order to make full use of the structural information in the graph and improve the emotional controllability of the comments, the present invention makes the following improvements to the traditional Transformer encoding and decoding architecture:
[0111] 2.1, Value Condition Embedding:
[0112] Map the preset value conditions (such as "positive", "optimistic") to Dimensional value condition embedding , as an additional input to the encoder, guiding the encoder to learn the scene graph embedding representation associated with the value condition. The calculation formula is:
[0113] ;
[0114] in, is the sentiment vocabulary query table, is a value condition (such as "positive", "optimistic").
[0115] 2.2, Scene graph encoding with emotional information:
[0116] The present invention further uses a graph neural network to evaluate the value of the scene graph Embedded representation learning is performed so that the encoding result H contains the semantic information of scene components and also integrates the value condition information. Tier The hidden state update formula of a node is:
[0117] ;
[0118] in, Representation Node The set of indexes of neighbor nodes; yes The elements in represent nodes Neighbor nodes The index of Indicates that in the l-1th layer, the node No. The hidden states of neighbor nodes; , For graph neural network The graph convolution parameter matrix of the layer is used to convert the hidden state of the nodes in the previous layer to the hidden state of the nodes in the current layer. For graph neural network The transformation matrix of the layer-value conditional embedding, is the activation function, Value point scene graph Midpoint and neighbor nodes adjacency matrix; by transforming the hidden states of neighboring nodes With the adjacency matrix and graph convolution parameters Multiply, aggregate the information of neighbor nodes into the hidden state update of the current node m, and use the connection relationship between nodes to transfer and aggregate information;
[0119] After L layers of graph convolution operations of the graph neural network, the encoding result H is obtained.
[0120] 2.3, Decoding stage value condition interaction:
[0121] In the decoding stage, in order to make the generated comments closer to the value condition constraints, the value conditions of the encoding stage are embedded It is also used as the input of the decoder. The Transformer decoder generates the final subjective review text word by word in an autoregressive manner, predicting one review word at a time. Let for The decoder hidden state at time t, for The comment words generated at the moment. The calculation formula is:
[0122] ;
[0123] It is a multi-head attention mechanism. The multi-head attention mechanism can fuse the encoding results in the decoder hidden state and sentiment conditional embedding , so that the generated comment words are aligned with the input video and meet the preset value conditions.
[0124] The conditional probability of the comment word output by the Transformer decoder is:
[0125] ;
[0126] The loss function of the autoencoder generation network adopts cross-extraction loss, while constraining the expressiveness of the value condition:
[0127] ;
[0128] Indicates time 0 Comments generated at every moment; The comment sentiment discriminator can determine whether the generated subjective comment text has the expected emotional tendency. In the present invention, the parameters of the comment sentiment discriminator are initialized using the pre-trained RoBERTa model. During the training process, the parameters of the comment sentiment discriminator and the autoencoder generation network are jointly updated through adversarial learning. Represents the conditional probability All The first term is the classic cross entropy loss, and the second term is the regularization term introduced by adversarial learning.
[0129] 3. Integrate adversarial learning guided by the value system.
[0130] In order to further improve the subjectivity and value guidance of reviews, the present invention introduces an adversarial learning mechanism based on the autoencoder generation network. The discriminator scores the generated reviews from the perspectives of sentimental tendency, value orientation, etc., and feeds the results back to optimize the generator, which is the autoencoder generation network mentioned above.
[0131] 3.1 Adversarial Learning under Value Conditions:
[0132] Let y be the subjective review text generated by the self-encoding network, and the corresponding value condition embedding is , is the real review text sampled from the real review dataset, and the corresponding value condition embedding is , the comment sentiment discriminator D is embedded with subjective comment text y and value condition For input, or with real comment text and value condition embedding As input, calculate the similarity between the input text sentiment and value condition embedding, determine the source of the subjective comment text and evaluate the sentiment tendency:
[0133] ; ;
[0134] Scoring function for the review sentiment discriminator.
[0135] Objective function of adversarial learning training for:
[0136] ;
[0137] That is, the self-encoding generation network G attempts to generate subjective comment text close to the real comment text to deceive the comment sentiment discriminator, while the comment sentiment discriminator D attempts to distinguish the real comment text from the subjective comment text. and They represent that the self-encoder generation network G should minimize the objective function and the comment sentiment discriminator D should maximize the objective function; represents the expected score of the comment sentiment discriminator D for the real comment text r, represents sampling of real review text from the distribution of real review text dataset, Represents the distribution of real review text dataset; represents the expected score of the comment sentiment discriminator D on the generated subjective comment text y, represents the generated review text sampled from the probability distribution output by the autoencoder generation network, Represents a self-encoding network to value the scene graph and value condition embedding Conditional distribution of review text for conditional prediction.
[0138] 3.2, Adversarial learning based on value system:
[0139] On the basis of the comment sentiment discriminator, a value evaluator Q is introduced to judge whether the subjective value orientation of the generated comment is consistent with the value system. Specifically, let P be a series of descriptive sentences or keywords that reflect the value system. The value evaluator Q first judges the real comment text Similarity with P :
[0140] ;
[0141] in, It is the feature extraction component of the value evaluator; then a gating mechanism is added, when the similarity is below the threshold When , the value evaluator Q outputs 0 to penalize comments with inappropriate value tendencies:
[0142] ;
[0143] The objective function of adversarial learning training of the comment generation model is replaced by :
[0144] .
[0145] Finally, the autoencoder generation network G can generate video review texts with subjective expression ability, positive emotional color and positive value orientation through continuous iterative optimization.
[0146] In the reasoning stage, for a given video input, the method of the present invention first extracts multimodal information and constructs a value perspective scene graph representation, and then generates a subjective comment text that matches the preset value condition (such as "positive" or "optimistic") through the self-encoding network prediction output.
[0147] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.
[0148] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.
Claims
1. A subjective review value editing method driven by a value system, which inputs a given video and preset value conditions into a trained review generation model and outputs a subjective review text that meets the value conditions. ; The training process of the review generation model includes the following steps: Step 1: Extract multimodal information of video V and obtain visual scene graph through visual modality sentiment analysis , through text modal value point of view mining to obtain text point of view graph , through the visual scene graph , Text Viewpoint Diagram And combined with the external value system knowledge graph , build a value perspective scene graph ; Step 2: View the scene graph from a value perspective As input, a self-encoding generation network consisting of an encoder and a Transformer decoder is used to generate subjective review text that matches the value conditions, including: Value condition embedding: Map the value condition to Dimensional value condition embedding ; Scene graph encoding with emotional information: The encoder uses a graph neural network based on Value point scene diagram Perform embedded representation learning to obtain the encoding result H that incorporates emotional information; Decoding stage value condition interaction: embed the encoding result H and value condition output by the encoder As the input of the Transformer decoder, the Transformer decoder generates a comment word at each moment in an autoregressive manner, and then obtains the final subjective comment text; Loss function of the autoencoder generation network Use cross entropy loss with regularization: ; is the conditional probability of the comment word output by the Transformer decoder, for Comments generated at all times, express The comment words generated before time, Indicates time 0 Comments generated at all times, is the comment sentiment discriminator, Represents the conditional probability All , is the weight coefficient.
2. According to the value system driven subjective review value editing method of claim 1, It is characterized in that It also includes introducing an adversarial learning mechanism in the training process of the review generation model, scoring the subjective review text generated by the autoencoder generation network through the review sentiment discriminator, and optimizing the autoencoder generation network through the scoring results, including: Let y be the subjective review text generated by the self-encoding network, and the corresponding value condition embedding is , is the real review text sampled from the real review dataset, and the corresponding value condition embedding is , the comment sentiment discriminator D is embedded with subjective comment text y and value condition For input, or with real comment text and value condition embedding As input, calculate the similarity between the generated subjective comment text or the real comment text and the value condition embedding, determine the source of the comment text and evaluate the emotional tendency: ; ; in, is the sigmoid activation function, Scoring function for comment sentiment discriminator; objective function for adversarial learning training of comment generation model for: ; That is, the autoencoder generation network G attempts to generate subjective review texts close to real review texts to deceive the review sentiment discriminator, while the review sentiment discriminator D attempts to distinguish between real review texts and subjective review texts, where and They represent that the self-encoder generation network G should minimize the objective function and the comment sentiment discriminator D should maximize the objective function; represents the expected score of the comment sentiment discriminator D for the real comment text r, represents sampling of real review text from the distribution of real review text dataset, Represents the distribution of real review text dataset; represents the expected score of the comment sentiment discriminator D on the generated subjective comment text y, represents the generated review text sampled from the probability distribution output by the autoencoder generation network, Represents a self-encoding network to value the scene graph and value condition embedding Conditional distribution for conditional prediction of subjective review text.
3. According to the value system driven subjective review value editing method of claim 2, It is characterized in that It also includes introducing a value evaluator Q based on the comment sentiment discriminator to calculate the value comment score, which is used to judge whether the subjective value orientation of the generated subjective comment text conforms to the value system, including: Let P be a series of descriptive sentences or keywords that reflect the value system. The value evaluator Q first calculates the real review text Similarity with P : ; in, It is the feature extraction component of the value evaluator; then a gating mechanism is added, when the similarity is below the threshold When the value evaluator Q output is 0, Score the final value review: ; The objective function of adversarial learning training of the comment generation model is replaced by : ; It indicates that the value evaluator Q and the comment sentiment discriminator D need to maximize the objective function.
4. According to the value system driven subjective review value editing method of claim 1, It is characterized in that In step 1, the extracting of multimodal information of the video V specifically includes: Using target detection technology and text extraction technology, we can obtain the visual object set in the video V and and a collection of text snippets , represents the nth visual object in the video V, represents the mth text segment in the video V; n and m represent the number of visual objects and the number of text segments in the video V, respectively.
5. According to the value system driven subjective review value editing method of claim 4, It is characterized in that In step 1, the visual scene graph obtained by visual modality sentiment analysis , specifically including: The visual objects in the video V detected by the target detection model are used as object nodes in the visual scene graph. The image sentiment classification model is used to identify the value tendency of the visual objects. The obtained sentiment probability vector is embedded into the visual scene graph as the node attribute of the object node. : ; ; in, is a collection of object nodes, Represents an object node and the sentiment probability vector Visual node tuple composed of A collection of Represents a connected object node The edge and spatial relationship representation of Visual edge tuples A collection of; For object nodes The emotion probability vector is the object node in the visual scene graph Node attributes of For the Object Nodes and Object Nodes The spatial relationship representation of is the edge attribute of the visual scene graph; .
6. According to the value system driven subjective review value editing method of claim 5, It is characterized in that In step 1, the text viewpoint graph is obtained by mining the text modal value viewpoints. , specifically including: The text segment is used as a text node in the text opinion graph; the sentiment intensity of the text node is extracted using a sentiment intensity regression model; Text nodes The emotional intensity for: ; in, is the sentiment intensity regression model, ; At the same time, using the topic model Extracting text nodes The topic distribution : ; Then use the sentiment dictionary to calculate the sentiment correlation scores between text nodes and each topic: ; in, Represents a text node With theme The sentiment-related score of For text nodes Each word in Sentiment score; For a given topic distribution Time, word Belong to the theme The probability of the text node sentiment intensity and topic sentiment score vector are used as the node attributes of the text node in the text opinion graph to construct the text opinion graph : ; ; in, is the kth text node The topic sentiment score vector contains all The sentiment score of each topic; is a collection of text nodes, Represents a text node , emotional intensity and the topic sentiment score vector A tuple of text nodes A collection of; For text nodes and text nodes The similarity of is the edge attribute of the text opinion graph; Represents a connected text node The edge and similarity Text edge tuples A collection of .
7. According to the value system driven subjective review value editing method of claim 6, It is characterized in that In step 1, the visual scene graph , Text Viewpoint Diagram And combined with the external value system knowledge graph , build a value perspective scene graph , specifically including: Use the externally provided value system knowledge graph as the value system knowledge graph , value system knowledge graph Each concept node in represents a value system concept in the value system knowledge graph; the value tendency score of each concept node is calculated: ; Concept Node The value orientation score is the concept node in the value system knowledge graph. The node properties of For the concept node in the value system knowledge graph A collection of connected concept nodes, for The concept node in Concept Node The importance weight of Concept Node With concept nodes Then, the visual scene graph , Text Viewpoint Diagram and value system knowledge graph The node attributes and edge attributes in the value point scene graph are embedded into middle: ; ; in, Value system knowledge graph The set of concept nodes in Respectively concept nodes and concept nodes, For object nodes With text nodes degree of relevance; Value system knowledge graph Medium Concept Node and concept nodes The relationship between is the value system knowledge graph Edge attributes of concept nodes , Concept Node and Composition of conceptual edge tuples ; A collection of tuples of nodes and node attributes representing the value point scene graph, including visual node tuples , text node tuple and a concept node tuple consisting of a concept node c and a value orientation score ; A set of tuples of edges and edge attributes representing the value point scene graph, including visual edge tuples , text edge tuple , by object node , text node and relevance The associated edge tuple , and the concept edge tuple .
8. According to the value system driven subjective review value editing method of claim 1, It is characterized in that In step 2, the encoder uses a graph neural network based on Value point scene diagram Perform embedded representation learning to obtain the encoding result H that incorporates emotional information, including: The hidden states of the nodes in the value scene graph are updated and learned through a multi-layer graph neural network to obtain the encoding result H that integrates the emotional information. Tier The hidden state of a node The update formula is: ; in, Representation Node The set of indexes of neighbor nodes; Is a node Neighbor nodes The index of Indicates that in the l-1th layer, the node No. The hidden states of neighbor nodes; , For graph neural network The graph convolution parameter matrix of the layer is used to convert the hidden state of the nodes in the previous layer to the hidden state of the nodes in the current layer. For graph neural network The transformation matrix of the layer-value conditional embedding, is the activation function, Value point scene graph Midpoint and neighbor nodes adjacency matrix; by transforming the hidden states of neighboring nodes With the adjacency matrix and the graph convolution parameter matrix Multiply, aggregate the information of neighbor nodes into the hidden state update of the current node, and use the connection relationship between nodes to transfer and aggregate information; After L layers of graph convolution operations of the graph neural network, the encoding result H is obtained.
9. According to the value system driven subjective review value editing method of claim 1, It is characterized in that The encoding result H output by the encoder and the value condition are embedded into As the input of the Transformer decoder, the Transformer decoder generates a comment word at each moment in an autoregressive manner, and then obtains the final subjective comment text, which includes: set up for The hidden state of the Transformer decoder at time t; The calculation formula is: ; in, is a multi-head attention mechanism, , , Represent the query matrix, key matrix and value matrix in the attention mechanism respectively; Conditional probability of review words output by Transformer decoder , represents the classification matrix, Represents the softmax function.
