Method, device and apparatus for generating event description text based on video data

Abstract

The present invention provides a method, apparatus, and device for generating event description text based on video data, which can be applied to the field of computer vision. The method comprises: constructing video scene graph data based on the video data to be processed, the video scene graph data comprising object nodes and edge relationships, wherein the object nodes represent human objects and objects related to the human objects in the video data to be processed; processing the video data to be processed based on a trained emotion classifier to obtain target emotion attribute nodes, wherein the target emotion attribute nodes represent the emotion attributes of the human objects; updating the video scene graph data based on the target emotion attribute nodes to obtain a video knowledge graph, wherein the video knowledge graph comprises edge relationships between the object nodes and the target emotion attribute nodes; and processing the video knowledge graph using a trained decoder to obtain target event description text used to describe the behavior of the human objects.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, device and equipment for generating event description text based on video data. Background Art

[0002] With the widespread adoption of video surveillance systems, intelligent video event analysis and description technologies have become a research hotspot in academia and industry. Traditional methods primarily focus on identifying the semantic content of events—detecting objects, behaviors, and their relationships within a video—while neglecting the emotional information inherent in the events. However, emotional factors play an indispensable role in understanding the full picture and characterizing the nature of an event. For example, crowds accompanied by anger often indicate potential group conflict, while celebratory crowds typically present no safety risks. Therefore, uncovering emotional cues in video events is crucial for comprehensively understanding the development of events and ensuring timely response and resolution. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method, apparatus and device for generating event description text based on video data.

[0004] According to a first aspect of the present invention, a method for generating event description text based on video data is provided, comprising: constructing video scene graph data based on the video data to be processed, the video scene graph data including object nodes and edge relationships, the object nodes representing the human objects and object objects related to the human objects in the video data to be processed; processing the video data to be processed based on a trained emotion classifier to obtain target emotion attribute nodes, the target emotion attribute nodes representing the emotion attributes of the human objects; updating the video scene graph data based on the target emotion attribute nodes to obtain a video knowledge graph, the video knowledge graph including edge relationships between object nodes and target emotion attribute nodes; extracting features of the target object nodes in the video knowledge graph to obtain node features; determining the i+1th description word probability distribution based on a generated description word sequence, target emotion attribute nodes and node features, the generated description word sequence including i description words, i≥0 and i is an integer; utilizing a decoding strategy to determine the description word corresponding to the maximum probability in the i+1th description word probability distribution as the i+1th description word; and obtaining the target event description text when the number of generated description words is greater than or equal to a preset number threshold.

[0005] According to an embodiment of the present invention, the video data to be processed is processed based on the trained emotion classifier to obtain the target emotion attribute node, including: using the visual feature extraction layer to process the video data to be processed to obtain visual features; using the audio feature extraction layer processor to process the video data to be processed to obtain audio features; processing the visual features based on the self-attention mechanism to obtain visual weighted features; performing feature fusion on the visual weighted features and the audio features to obtain multimodal emotion features; performing emotion attribute classification on the multimodal emotion features to obtain the target emotion attribute node.

[0006] According to an embodiment of the present invention, the trained emotion classifier is determined based on the following operations: obtaining video training data and emotion category label data; using an activation function to process multimodal emotion training features to obtain a predicted emotion probability distribution of the emotion category to which the video belongs, and the multimodal emotion training features are obtained based on the video training data; determining a loss function based on the predicted emotion probability distribution and the emotion category label data; minimizing the loss function based on a stochastic gradient descent algorithm to obtain a trained emotion classifier.

[0007] According to an embodiment of the present invention, the trained decoder is trained based on the following steps: obtaining training event description text, training object nodes and training emotion attribute nodes; using a graph convolutional neural network to process the training object nodes to obtain training node context features; processing the training node context features and event description text features according to a multi-layer perceptron to obtain the mean and variance, and the event description text features are generated based on the training event description text; using a decoder to process the description word sequence, training object nodes and training emotion attribute nodes generated before the tth moment to obtain the description word probability distribution of the training event description text at the tth moment, where t is an integer greater than zero; determining the variational lower bound based on the description word probability distribution, mean and variance; maximizing the variational lower bound to obtain the trained decoder.

[0008] According to an embodiment of the present invention, a graph convolutional neural network is used to process training object nodes to obtain contextual features of the training nodes, including: determining the n-th layer node representation matrix based on the n-1th layer node representation matrix of the graph convolutional neural network; processing the n-th layer training object node representation matrix and the n-th layer other node representation matrix according to the activation function to obtain the attention weight of the training object node to other nodes, wherein the training object node corresponds to the n-th layer training object node representation matrix, and the other nodes correspond to the n-th layer other node representation matrix; obtaining the training node contextual features based on the attention weight and the n-th layer other node representation matrix, where n is an integer greater than 1.

[0009] According to an embodiment of the present invention, a decoder is used to process the description word sequence, training object nodes and training emotion attribute nodes generated before the t moment to obtain the description word probability distribution of the training event description text at the t moment, including: using a decoder to process the description word sequence, training emotion attribute nodes and training object nodes generated before the t moment to obtain the decoded hidden layer representation features at the t moment; using a normalization algorithm to process the decoded hidden layer representation features at the t moment to obtain the description word probability distribution of the training event description text at the t moment.

[0010] The second aspect of the present invention provides an event description text generation device based on video data, including: a construction module for constructing video scene graph data based on the video data to be processed, the video scene graph data including object nodes and edge relationships, the object nodes representing the character objects and object objects related to the character objects in the video data to be processed; a target emotion attribute node acquisition module for processing the video data to be processed based on the trained emotion classifier to obtain the target emotion attribute node, the target emotion attribute node representing the emotion attribute of the character object; a video knowledge graph acquisition module for updating the video scene graph data based on the target emotion attribute node to obtain the video knowledge graph, the video knowledge graph including the object nodes and the target emotion attribute node. The edge relationship between the emotion attribute nodes; the node feature acquisition module is used to extract the features of the target object nodes in the video knowledge graph to obtain the node features; the description word probability distribution module is used to determine the probability distribution of the i+1th description word based on the generated description word sequence, the target emotion attribute node and the node features, and the generated description word sequence includes i description words, i≥0 and i is an integer; the description word determination module is used to use the decoding strategy to determine the description word corresponding to the maximum probability in the i+1th description word probability distribution as the i+1th description word; the target event description text acquisition module is used to obtain the target event description text when the number of generated description words is greater than or equal to the preset number threshold.

[0011] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0012] The fourth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0013] According to an embodiment of the present invention, by constructing a video scene graph based on the video data to be processed, the video data to be processed is processed based on the trained emotion classifier to obtain a target emotion attribute node. The addition of the target emotion attribute node makes the understanding of video surveillance no longer limited to the visual level, but can also capture the emotional atmosphere in the scene, thereby more comprehensively reflecting the nature of the event. The video scene graph data is updated based on the target emotion attribute node to obtain a video knowledge graph, which can better understand the dynamic events in the video. This dynamic understanding ability can adapt to complex and changeable monitoring scenes and can detect abnormal emotions or potential risks in time. The trained decoder is used to process the video knowledge graph to obtain a target event description text for describing the behavior of the character object. The target event description text has a target emotion attribute. By introducing the target emotion attribute node, the decoded text can not only accurately portray the content of the event, but also reflect the subtle emotional color, so as to better judge the risk level and emergency status of the event. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0015] Figure 1 A diagram showing an application scenario of a method and apparatus for generating event description text based on video data according to an embodiment of the present invention;

[0016] Figure 2 A flowchart of a method for generating event description text based on video data according to an embodiment of the present invention is shown;

[0017] Figure 3 A schematic diagram showing a method for generating event description text based on video data according to an embodiment of the present invention is shown;

[0018] Figure 4 It shows a structural block diagram of a device for generating event description text based on video data according to an embodiment of the present invention;

[0019] Figure 5 A block diagram of an electronic device suitable for implementing a method for generating event description text based on video data according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0020] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0021] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0022] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0023] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0024] Traditional methods primarily focus on identifying the semantic content of events—detecting objects, behaviors, and their relationships within a video—while neglecting the emotional information inherent in the event. However, emotional factors play an indispensable role in understanding the full picture and characterizing the nature of an event. For example, crowd gatherings accompanied by anger often indicate a potential risk of group conflict, while crowds in a celebratory atmosphere typically present no safety concerns. Therefore, mining emotional cues within video events is crucial for comprehensively understanding the development of events and ensuring timely response and disposal. However, current research largely focuses on identifying event types, while ignoring the value of emotional semantics for generating event descriptions. These issues hinder the development of emotion-driven intelligent video event analysis technology.

[0025] In view of this, the present invention provides a method for generating event description text based on video data, and an apparatus and device for generating event description text based on video data. The method comprises: constructing video scene graph data based on the video data to be processed, the video scene graph data comprising object nodes and edge relationships, wherein the object nodes represent human objects and objects related to the human objects in the video data to be processed; processing the video data to be processed based on a trained emotion classifier to obtain target emotion attribute nodes, wherein the target emotion attribute nodes represent the emotion attributes of the human objects; updating the video scene graph data based on the target emotion attribute nodes to obtain a video knowledge graph, wherein the video knowledge graph comprises edge relationships between object nodes and target emotion attribute nodes; and processing the video knowledge graph using a trained decoder to obtain target event description text for describing the behavior of the human objects.

[0026] It should be noted that the event description text generation method based on video data and the event description text generation device based on video data provided by the present invention can be used in the field of computer vision, and can also be used in any field other than the field of computer vision, such as the field of urban transportation. Therefore, the application field of the event description text generation method based on video data and the event description text generation device based on video data provided by the present invention is not limited.

[0027] In the technical solution of the present invention, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] In scenarios where personal information is used for automated decision-making, the methods, devices, and systems provided by embodiments of the present invention provide users with corresponding operational portals, allowing them to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The term "automated decision-making" herein refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, or credit status through computer programs and making decisions. The term "expert decision-making" herein refers to the activity of decision-making by individuals who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0029] Figure 1 A diagram illustrating an application scenario of a method and apparatus for generating event description text based on video data according to an embodiment of the present invention is shown.

[0030] like Figure 1 As shown, the application scenario according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables, etc.

[0031] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0032] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0034] It should be noted that the event description text generation method based on video data provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the event description text generation device based on video data provided in the embodiment of the present invention can generally be set in the server 105. The event description text generation method based on video data provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the event description text generation device based on video data provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0036] Figure 2 A flowchart of a method for generating event description text based on video data according to an embodiment of the present invention is shown.

[0037] like Figure 2 As shown, the method for generating an event description text based on video data of this embodiment includes operations S210 to S270. The method for generating an event description text based on video data can be executed by an electronic device.

[0038] In operation S210 , video scene graph data is constructed based on the video data to be processed, where the video scene graph data includes object nodes and edge relationships.

[0039] In operation S220 , the video data to be processed is processed based on the trained emotion classifier to obtain a target emotion attribute node, where the target emotion attribute node represents an emotion attribute of a person object.

[0040] In operation S230 , the video scene graph data is updated based on the target emotion attribute node to obtain a video knowledge graph, which includes edge relationships between object nodes and target emotion attribute nodes.

[0041] In operation S240 , feature extraction is performed on the target object node in the video knowledge graph to obtain node features.

[0042] In operation S250 , the probability distribution of the i+1th description word is determined based on the generated description word sequence, the target sentiment attribute node, and the node features, where the generated description word sequence includes i description words, i≥0 and i is an integer.

[0043] In operation S260 , a decoding strategy is used to determine the description word corresponding to the maximum probability in the (i+1)th description word probability distribution as the (i+1)th description word.

[0044] In operation S270 , when the number of generated description words is greater than or equal to a preset number threshold, a target event description text is obtained.

[0045] According to an embodiment of the present invention, video scene graph data can be constructed based on the video data to be processed. The video scene graph data can include object nodes and edge relationships. The object nodes represent human objects and object objects related to the human objects in the video data to be processed. For example, in a video scene graph, there are "person," "football," and "kick." Among them, "person" and "football" can be represented as object nodes, and "kick" can be represented as an edge relationship.

[0046] According to an embodiment of the present invention, an existing video scene graph data generation algorithm (such as RelTR, etc.) can be used to construct a scene graph for each segment of video data to be processed. ,in , Represents a collection of object nodes in the scene, Indicates the object nodes, Indicates the total number of objects in the video to be processed. Represents the set of relationship edges between object nodes, Describe object node and object nodes The type of relationship between .

[0047] According to an embodiment of the present invention, a trained emotion classifier is used to process the video data to obtain a target emotion attribute node, which represents the emotion attribute of a person. Emotion attributes may include, but are not limited to, emotional states such as happiness, sadness, anger, and fear, and are not limited thereto. The embodiments of the present invention are not limited thereto.

[0048] According to an embodiment of the present invention, in order to integrate the overall emotional information of the video data to be processed, the scene graph constructed Introducing sentiment attribute nodes .remember for dimensional emotional attribute node embedding vector. The video knowledge graph includes the edge relationship between the object node and the target emotional attribute node. Based on the target emotional attribute node, the video scene graph data is updated to obtain the video knowledge graph. Finally, the emotion-enhanced video knowledge graph is obtained. ,in Associating edges with newly added emotional attribute nodes, reflecting Dependencies with other nodes.

[0049] According to an embodiment of the present invention, by updating the video scene graph data based on the target emotion attribute node, a video knowledge graph can be obtained, and the video knowledge graph can include edge relationships between object nodes and target emotion attribute nodes. Introducing emotion attribute nodes on the basis of traditional video scene graphs enhances the depiction of the emotional semantics of events by graph representation. The emotion-enhanced scene event graph integrates rich audiovisual clues, emotional information, and temporal associations between events, providing a powerful knowledge representation for a deep understanding of video events. At the same time, the introduction of emotion nodes as latent variables in the generative model enables the decoded text to not only accurately depict the content of the event, but also reflect subtle emotional colors. This description, which is full of emotion and consistent with facts, plays an important role in helping managers fully grasp the status of video events and make timely and effective decision responses.

[0050] According to embodiments of the present invention, a trained decoder is used to process a video knowledge graph to obtain target event description text describing the behavior of characters, objects, and behaviors. The decoder can use a Recurrent Neural Network (RNN)-based decoder, a Transformer-based decoder, a large language model-based decoder, and other methods, but is not limited to these methods and is not limited to these methods by the embodiments of the present invention. A video knowledge graph can include multiple characters and different emotion attribute nodes. For example, an initial video scene graph contains "person," "football," and "kick." "person" and "football" can be represented as object nodes, and "kick" can be represented as an edge relationship. The video data is processed using a trained emotion classifier to obtain a target emotion attribute node with the attribute "happy." Based on this target emotion attribute node, the initial video scene graph data is updated to obtain a video knowledge graph. The trained decoder is used to process the video knowledge graph to obtain the target event description text "A person is happily kicking a football."

[0051] According to an embodiment of the present invention, by constructing a video scene graph based on the video data to be processed, the video data to be processed is processed based on the trained emotion classifier to obtain the target emotion attribute node. The addition of the target emotion attribute node makes the understanding of video surveillance no longer limited to the visual level, but can also capture the emotional atmosphere in the scene, thereby more comprehensively reflecting the nature of the event. The video scene graph data is updated based on the target emotion attribute node to obtain a video knowledge graph, which can better understand the dynamic events in the video. This dynamic understanding ability can adapt to complex and changeable monitoring scenes and can timely detect abnormal emotions or potential risks. The trained decoder is used to process the video knowledge graph to obtain a target event description text for describing the behavior of the character object. The target event description text has the target emotion attribute. By introducing the target emotion attribute node, the decoded text can not only accurately depict the content of the event, but also reflect the subtle emotional color, better judge the risk level and emergency state of the event, make up for the shortcomings of single modal analysis and the shortcomings of ignoring emotional factors, and provide new ideas for tasks such as smart security and decision support.

[0052] According to an embodiment of the present invention, the video data to be processed is processed based on the trained emotion classifier to obtain the target emotion attribute node, including: using the visual feature extraction layer to process the video data to be processed to obtain visual features; using the audio feature extraction layer processor to process the video data to be processed to obtain audio features; processing the visual features based on the self-attention mechanism to obtain visual weighted features; performing feature fusion on the visual weighted features and the audio features to obtain multimodal emotion features; performing emotion attribute classification on the multimodal emotion features to obtain the target emotion attribute node.

[0053] According to an embodiment of the present invention, visual features can be obtained by processing the video data to be processed using a visual feature extraction layer. The visual feature extraction layer can utilize a vision transformer (ViT), an efficient neural network (EfficientNet), or the like. Simultaneously, an audio feature extraction layer processor can be used to process the video data to obtain audio features. The audio feature extraction layer can utilize a VGG-like audio feature extractor (VGGish). In this embodiment, the EfficientNet-B5 model (EfficientNet-B5) can be selected as the visual feature extraction layer, outputting a 2048-dimensional visual feature vector; the VGGish network extracts a 128-dimensional audio feature vector.

[0054] According to an embodiment of the present invention, visual features are processed based on the self-attention mechanism to obtain visual weighted features, as specifically shown in formula (1).

[0055] (1);

[0056] Among them, Q represents the query matrix of visual features, Key represents the key matrix of visual features, and V represents the value matrix of visual features. Represents the visual weighted features, and i represents the i-th video in the video data to be processed.

[0057] According to an embodiment of the present invention, frames with higher relevance can obtain greater weights, and a multi-head attention structure can be adopted. The number of heads can be set to 8, but is not limited to this. The embodiment of the present invention does not limit the number of heads.

[0058] According to an embodiment of the present invention, by performing feature fusion on visual weighted features and audio features, multimodal emotion features can be obtained, as specifically shown in formula (2).

[0059] (2);

[0060] in, represents the multimodal emotional features represented by the i-th video, Represents vector concatenation operation, represents a multilayer perceptron network, represents the visual weighted features of the i-th video in the video data to be processed, Represents the audio features corresponding to the i-th video.

[0061] According to an embodiment of the present invention, by performing emotional attribute classification on the multimodal emotional features, a target emotional attribute node can be obtained, as specifically shown in formula (3).

[0062] (3);

[0063] in, represents the target emotional attribute node corresponding to the i-th video, W is the emotional attribute node embedding matrix, Represents the multimodal emotional features represented by the i-th video.

[0064] According to embodiments of the present invention, by combining visual and audio features, emotional information in videos can be more comprehensively captured. While a single modality (such as audio or visual alone) may not fully convey emotion, multimodal fusion can overcome the shortcomings of a single modality and improve the accuracy of emotion classification.

[0065] According to an embodiment of the present invention, a trained decoder is used to process a video knowledge graph to obtain a target event description text for describing the behavior of a character object, including: extracting features of the target object node in the video knowledge graph to obtain node features; determining the i+1th description word probability distribution based on the generated description word sequence, target emotional attribute node and node features, the generated description word sequence includes i description words, i≥0 and i is an integer; using a decoding strategy, determining the description word corresponding to the maximum probability in the i+1th description word probability distribution as the i+1th description word; and obtaining the target event description text when the number of generated description words is greater than or equal to a preset number threshold.

[0066] According to an embodiment of the present invention, feature extraction is performed on target object nodes in a video knowledge graph to obtain node features.

[0067] According to an embodiment of the present invention, the probability distribution of the i+1th description word can be determined based on the generated description word sequence, the target sentiment attribute node, and the node features. The generated description word sequence may include i description words, where i ≥ 0 and i is an integer. The generated description word sequence, the target sentiment attribute node, and the node features can be processed using a Transformer decoder to determine the probability distribution of the i+1th description word, as shown in Formula (4).

[0068] (4);

[0069] in, Indicates time The description word sequence generated previously, x represents the node feature, z represents the target emotional attribute node, represents the probability distribution of the i+1th description word, y represents the target event description text, Represents the parameters involved in the loss function, and T represents a total of T moments.

[0070] According to an embodiment of the present invention, the decoder can be used to determine the description word corresponding to the maximum probability in the probability distribution of the i+1th description word as the i+1th description word, as shown in formula (5).

[0071] (5);

[0072] in, Indicates the descriptive word with the highest probability in the target event description text, Indicates the event description text sequence generated at time t. It represents the description word generated at the predicted t+1th moment, that is, the i+1th description word mentioned above, x represents the node feature, and z represents the target sentiment attribute node.

[0073] According to an embodiment of the present invention, when the number of generated description words is greater than or equal to a preset number threshold, a target event description text can be obtained. , which means a sequence starting with the start symbol. Repeat the following steps until terminator to get the target event description text.

[0074] According to an embodiment of the present invention, a decoder can be used to describe events, embedding features corresponding to emotion-rich emotion attribute nodes into the decoding process. By introducing emotion attributes, the system can better judge the risk level and emergency status of an event. Including emotion as a latent variable in the decoding process ensures that the generated event description text not only conforms to the actual content of the event but also reflects the corresponding emotional tone.

[0075] According to an embodiment of the present invention, the trained emotion classifier is determined based on the following operations: obtaining video training data and emotion category label data; using an activation function to process multimodal emotion training features to obtain a predicted emotion probability distribution of the emotion category to which the video belongs, and the multimodal emotion training features are obtained based on the video training data; determining a loss function based on the predicted emotion probability distribution and the emotion category label data; minimizing the loss function based on a stochastic gradient descent algorithm to obtain a trained emotion classifier.

[0076] According to an embodiment of the present invention, video training data and emotion category label data can be obtained. Video training data covering different scenes and event types can be collected from multiple channels such as video websites and monitoring systems. To enrich the emotional diversity of the data, various emotional states such as happiness, sadness, anger, fear, disgust, and surprise are included as much as possible. In this embodiment, a total of approximately 10,000 video clips can be collected, with each clip lasting between 10 seconds and 5 minutes.

[0077] According to an embodiment of the present invention, multiple professionals can watch each video, subjectively judge the emotion category and intensity based on the video content, and make corresponding annotations, ultimately obtaining emotion category label data.

[0078] According to an embodiment of the present invention, the multimodal emotion training features are obtained based on video training data. By processing the multimodal emotion training features using an activation function, the predicted emotion probability distribution of the emotion category to which the video belongs can be obtained, as shown in Formula (6).

[0079] (6);

[0080] in, represents the predicted emotional probability distribution of the emotional category of the i-th video, Softmax represents the activation function, The embedding matrix representing the total number of emotion categories C, are the classifier parameters, Represents the multimodal emotional features represented by the i-th video.

[0081] According to an embodiment of the present invention, a loss function may be determined based on the predicted emotion probability distribution and emotion category label data, as specifically shown in formula (7).

[0082] (7);

[0083] in, represents the loss function, is the total number of preset emotion categories, N is the total number of videos, represents the predicted emotional probability distribution of the emotional category j of the i-th video, Indicates the emotional category label of the i-th video Dimension value.

[0084] According to an embodiment of the present invention, the parameters involved in the loss function can be recorded as .

[0085] According to an embodiment of the present invention, the loss function is processed according to the stochastic gradient descent algorithm, the learning rate can be set to 0.0001, the number of training iterations can be 100, and the batch size can be 32. Finally, a trained sentiment classifier can be obtained, as specifically shown in formula (8).

[0086] (8);

[0087] in, represents the trained sentiment classifier, Represents the parameters involved in minimizing the loss function , represents the loss function.

[0088] According to an embodiment of the present invention, by training with multimodal emotion features and emotion annotation data, an efficient and accurate video emotion classifier is established, which provides key emotion recognition capabilities for subsequent event analysis. The emotion classifier can automatically process large amounts of video data and quickly output emotion attribute nodes, saving time and cost for manual annotation. According to an embodiment of the present invention, the trained decoder is trained based on the following steps: obtaining training event description text, training object nodes, and training emotion attribute nodes; processing the training object nodes using a graph convolutional neural network to obtain training node context features; processing the training node context features and event description text features according to a multi-layer perceptron to obtain mean and variance, and the event description text features are generated based on the training event description text; processing the description word sequence, training object nodes, and training emotion attribute nodes generated before the tth moment using a decoder to obtain the description word probability distribution of the training event description text at the tth moment, where t is an integer greater than zero; determining the variational lower bound based on the description word probability distribution, mean, and variance; maximizing the variational lower bound to obtain a trained decoder.

[0089] According to an embodiment of the present invention, the decoder can be composed of two parts: an inference network and a generation network. It can obtain training event description text, training object nodes, and training sentiment attribute nodes. A graph convolutional neural network is used to process the training object nodes to obtain training node context features.

[0090] According to an embodiment of the present invention, event description text features are generated based on training event description text. The mean and variance can be obtained by processing the training node context features and event description text features using a multilayer perceptron (MLP) network, as shown in formula (9):

[0091] (9);

[0092] in, represents a multilayer perceptron network, represents the contextual features of the training node, Indicates that there are training nodes related to the emotional attribute node e in the i-th video. indivual, Indicates the text features of the event description. represents the mean, Represents variance.

[0093] According to an embodiment of the present invention, the description word sequence, training object node and training emotion attribute node generated before the tth moment are processed by the decoder to obtain the description word probability distribution of the training event description text at the tth moment, where t is an integer greater than zero.

[0094] According to an embodiment of the present invention, the variational lower bound is determined based on the probability distribution, mean, and variance of the description words, as shown in formulas (10), (11), (12), and (13).

[0095] (10);

[0096] Among them, ELBO represents the variational lower bound, represents the parameters involved in minimizing the loss function in the generative network, Represents the reconstruction loss, that is, Under the sampled target sentiment attribute node z, the generative network reconstructs the log-likelihood of the target event description text y. The second term is the posterior distribution With prior distribution The information divergence (Kullback-Leibler Divergence, KL) between is used to constrain the learned posterior to be as close to the prior as possible, Represents the parameters involved in the inference network.

[0097] In practice, the reconstruction error term can be rewritten as:

[0098] (11);

[0099] Among them, L represents the number of samples of sentiment attribute nodes.

[0100] According to an embodiment of the present invention, the prior distribution It can be assumed to be a standard Gaussian distribution ,and The KL divergence between can be obtained analytically:

[0101] (12);

[0102] in, It can represent the mean of the i-th video, It can represent the variance of the i-th video.

[0103] In summary, the variational lower bound (10) can be approximately expressed as:

[0104] (13);

[0105] Among them, ELBO represents the variational lower bound, It can represent the mean of the i-th video, It can represent the variance of the i-th video.

[0106] According to an embodiment of the present invention, the variational lower bound is maximized to obtain a trained decoder.

[0107] According to an embodiment of the present invention, during model training, an Adaptive Moment Estimation (Adam) optimizer can be used to maximize the Evidence Lower Bound (ELBO). The learning rate can be set to 0.0005, the number of training rounds can be 150, and the batch size can be 48. A video dataset can be constructed containing 1,000 video samples, divided into training, validation, and test sets at an 8:1:1 ratio. Finally, the content quality of the generated text can be evaluated using the automated Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and the Metric for Evaluation of Translation with Explicit Ordering (METEOR) metrics. Experimental results show that the corresponding system of the present invention achieves Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence (ROUGE-L) and METEOR values ​​of 35.2 and 27.5, respectively, on the video dataset. Qualitative analysis also confirms that the event description text generated by the method of the present invention better depicts the emotional semantics of the video event, and the description is richer and more vivid.

[0108] According to an embodiment of the present invention, a graph convolutional neural network is used to process training object nodes to obtain contextual features of the training nodes, including: determining the n-th layer node representation matrix based on the n-1th layer node representation matrix of the graph convolutional neural network; processing the n-th layer training object node representation matrix and the n-th layer other node representation matrix according to the activation function to obtain the attention weight of the training object node to other nodes, wherein the training object node corresponds to the n-th layer training object node representation matrix, and the other nodes correspond to the n-th layer other node representation matrix; obtaining the training node contextual features based on the attention weight and the n-th layer other node representation matrix, where n is an integer greater than 1.

[0109] According to an embodiment of the present invention, the node representation matrix of the nth layer can be determined based on the node representation matrix of the n-1th layer of the graph convolutional neural network, as shown in formula (14).

[0110] (14);

[0111] in, is the n-th layer node representation matrix, represents the n-1th layer node representation matrix, is the initial representation matrix of the node, represents the correlation function, A matrix representing the adjacency matrix of the video knowledge graph plus the self-connection matrix, for The degree matrix of are trainable weights.

[0112] According to an embodiment of the present invention, in addition to information aggregation at the node level, the interaction relationship between nodes is also required, especially to capture the dependency between emotion nodes and object nodes. The training object node corresponds to the n-th layer training object node representation matrix, and other nodes correspond to the n-th layer other node representation matrix. The n-th layer training object node representation matrix and the n-th layer other node representation matrix can be processed according to the activation function to obtain the attention weight of the training object node on other nodes, as shown in formula (15):

[0113] (15);

[0114] Among them, Softmax represents the activation function, Indicates the training object node K’s effect on other nodes The attention weight, Represents the n-th layer training object node K represents the matrix, Indicates other nodes in the nth layer represents the matrix, and is the parameter matrix of the attention network.

[0115] According to an embodiment of the present invention, the context feature of the training node can be obtained based on the attention weight and the representation matrix of other nodes in the nth layer, as shown in formula (16).

[0116] (16);

[0117] in, represents the contextual features of the training node, represents the attention weight, Indicates other nodes in the nth layer represents the matrix, Represents the parameter matrix of the attention network.

[0118] According to an embodiment of the present invention, by using graph neural network modeling, the causal logic and emotional transfer patterns between events are mined. The graph neural network is used to enable information exchange between events, where each node not only represents a person or object but also carries emotional information.

[0119] According to an embodiment of the present invention, a decoder is used to process the description word sequence, training object nodes and training emotion attribute nodes generated before the t moment to obtain the description word probability distribution of the training event description text at the t moment, including: using a decoder to process the description word sequence, training emotion attribute nodes and training object nodes generated before the t moment to obtain the decoded hidden layer representation features at the t moment; using a normalization algorithm to process the decoded hidden layer representation features at the t moment to obtain the description word probability distribution of the training event description text at the t moment.

[0120] According to an embodiment of the present invention, a decoder is used to process the description word sequence, training emotion attribute nodes and training object nodes generated before the tth moment to obtain the decoded hidden layer representation features at the tth moment, as specifically shown in formula (17).

[0121] (17)

[0122] in, represents the decoding hidden layer representation feature at time t, represents the parameters involved in minimizing the loss function in the generative network, Indicates time The description word sequence generated previously, To train the sentiment attribute nodes, Represents the representation matrix of the L-th layer training object node K, Indicates that there are training nodes related to the emotional attribute node e in the i-th video. indivual.

[0123] According to an embodiment of the present invention, a normalization algorithm is used to process the decoded hidden layer representation features at the t-th moment to obtain the probability distribution of the description words of the training event description text at the t-th moment, as specifically shown in formula (18).

[0124] (18);

[0125] in, represents the probability distribution of the description words of the training event description text at time t, Indicates the decoding hidden layer representation feature at time t, Softmax represents the activation function, and Represents the parameter matrix in the decoder.

[0126] Figure 3 A schematic diagram of a method for generating event description text based on video data according to an embodiment of the present invention is shown.

[0127] like Figure 3 As shown, the event description text generation method based on video data includes three steps, namely, emotion attribute node generation 310, video knowledge graph construction 320 and event description text generation 330. Specifically, emotion attribute node generation 310 includes: obtaining the video surveillance data to be processed, inputting the video data to be processed into the audio feature extractor to output audio features, and inputting the video data to be processed into the visual feature extractor to output visual features. Based on the visual features and audio features, multimodal emotion features can be obtained. The multimodal emotion features are input into the emotion classifier to output emotion attribute nodes. Video knowledge graph construction 320 includes: video scene graph data can be constructed based on the video data to be processed. The emotion attribute nodes and video scene graph data output by emotion attribute node generation 310 are updated using a graph neural network to obtain video scene graph data. Event description text generation 330 includes: using a decoder to process the target emotion attribute node and video knowledge graph to obtain the target event description text.

[0128] Figure 4 The structure block diagram of the device for generating event description text based on video data according to an embodiment of the present invention is shown.

[0129] like Figure 4 As shown, the event description text generation device based on video data of this embodiment includes a construction module 410, a target emotion attribute node acquisition module 420, a video knowledge graph acquisition module 430, a node feature acquisition module 440, a description word probability distribution module 450, a description word determination module 460 and a target event description text acquisition module 470.

[0130] The construction module 410 is used to process the video data to be processed based on the trained emotion classifier to obtain a target emotion attribute node, where the target emotion attribute node represents the emotion attribute of the person object.

[0131] The target sentiment attribute node obtaining module 420 is used to process each traffic sequence data and topology graph using a graph neural network to obtain traffic spatiotemporal characteristics.

[0132] The video knowledge graph obtaining module 430 is used to update the video scene graph data based on the target emotion attribute node to obtain the video knowledge graph, which includes the edge relationship between the object node and the target emotion attribute node.

[0133] The node feature acquisition module 440 is used to extract features of the target object nodes in the video knowledge graph to obtain node features.

[0134] The description word probability distribution module 450 is used to determine the probability distribution of the i+1th description word based on the generated description word sequence, target sentiment attribute node and node features. The generated description word sequence includes i description words, i≥0 and i is an integer.

[0135] The description word determination module 460 is configured to use a decoding strategy to determine the description word corresponding to the maximum probability in the probability distribution of the (i+1)th description word as the (i+1)th description word.

[0136] The target event description text obtaining module 470 is configured to obtain the target event description text when the number of generated description words is greater than or equal to a preset number threshold.

[0137] According to an embodiment of the present invention, by constructing a video scene graph based on the video data to be processed, the video data to be processed is processed based on the trained emotion classifier to obtain a target emotion attribute node. The addition of the target emotion attribute node makes the understanding of video surveillance no longer limited to the visual level, but can also capture the emotional atmosphere in the scene, thereby more comprehensively reflecting the nature of the event. The video scene graph data is updated based on the target emotion attribute node to obtain a video knowledge graph, which can better understand the dynamic events in the video. This dynamic understanding ability can adapt to complex and changing monitoring scenes, and can timely detect abnormal emotions or potential risks. The trained decoder is used to process the video knowledge graph to obtain a target event description text for describing the behavior of the character object. The target event description text has a target emotion attribute. By introducing the target emotion attribute node, the decoded text can not only accurately portray the content of the event, but also reflect the subtle emotional color, and better judge the risk level and emergency status of the event.

[0138] According to an embodiment of the present invention, the target emotion attribute node obtaining module 420 includes: a visual feature obtaining unit, an audio feature obtaining unit, a visual weighted feature obtaining unit, a feature fusion unit and an emotion attribute classification unit.

[0139] The visual feature obtaining unit is used to process the video data to be processed using the visual feature extraction layer to obtain visual features.

[0140] The audio feature obtaining unit is used to process the video data to be processed using the audio feature extraction layer processor to obtain audio features.

[0141] The visual weighted feature acquisition unit is used to process visual features based on the self-attention mechanism to obtain visual weighted features.

[0142] The feature fusion unit is used to fuse the visual weighted features and audio features to obtain multimodal emotional features.

[0143] The emotion attribute classification unit is used to classify the emotion attributes of the multimodal emotion features and obtain the target emotion attribute node.

[0144] According to an embodiment of the present invention, the target emotion attribute node obtaining module 420 includes: a first obtaining unit, an emotion probability distribution prediction unit, a loss function obtaining unit and a first training unit.

[0145] The first acquisition unit is used to acquire video training data and emotion category label data.

[0146] The predicted emotion probability distribution unit is used to process the multimodal emotion training features using the activation function to obtain the predicted emotion probability distribution of the emotion category to which the video belongs. The multimodal emotion training features are obtained based on the video training data.

[0147] The loss function obtaining unit is used to process the predicted emotion probability distribution and emotion category label data using the cross entropy loss function to obtain the loss function.

[0148] The first training unit is used to process the loss function according to the stochastic gradient descent algorithm to obtain a trained sentiment classifier.

[0149] According to an embodiment of the present invention, the video knowledge graph obtaining module 430 includes: a second acquisition unit, a training node context feature obtaining unit, a mean variance obtaining unit, a description word probability distribution unit, a variational lower bound determination unit and a second training unit.

[0150] The second acquisition unit is used to acquire the training event description text, the training object node and the training emotion attribute node.

[0151] The training node context feature obtaining unit is used to process the training object nodes using a graph convolutional neural network to obtain the training node context features.

[0152] The mean-variance obtaining unit is used to process the training node context features and event description text features according to the multi-layer perceptron to obtain the mean and variance. The event description text features are generated based on the training event description text.

[0153] The description word probability distribution unit is used to use the decoder to process the description word sequence, training object node and training emotion attribute node generated before the tth moment to obtain the description word probability distribution of the training event description text at the tth moment, where t is an integer greater than zero.

[0154] The variational lower bound determination unit is used to determine the variational lower bound according to the probability distribution, mean and variance of the description words.

[0155] The second training unit is used to maximize the variational lower bound to obtain the trained decoder.

[0156] According to an embodiment of the present invention, the training node context feature obtaining unit includes: a node representation matrix determination subunit, an attention weight obtaining subunit and a training node context feature obtaining subunit.

[0157] The node representation matrix determination subunit is used to determine the node representation matrix of the nth layer according to the node representation matrix of the n-1th layer of the graph convolutional neural network.

[0158] The attention weight obtaining subunit is used to process the n-th layer training object node representation matrix and the n-th layer other node representation matrix according to the activation function to obtain the attention weight of the training object node to the other nodes, wherein the training object node corresponds to the n-th layer training object node representation matrix, and the other nodes correspond to the n-th layer other node representation matrix.

[0159] The training node context feature obtaining subunit is used to obtain the training node context feature according to the attention weight and the representation matrix of other nodes in the nth layer.

[0160] According to an embodiment of the present invention, the description word probability distribution unit includes: a decoding hidden layer representation feature subunit and a description word probability distribution subunit.

[0161] The decoding hidden layer representation feature subunit is used to use the decoder to process the description word sequence, training emotion attribute node and training object node generated before the tth moment to obtain the decoding hidden layer representation feature at the tth moment.

[0162] The description word probability distribution subunit is used to process the decoded hidden layer representation features at the t-th moment using a normalization algorithm to obtain the description word probability distribution of the training event description text at the t-th moment.

[0163] According to an embodiment of the present invention, any multiple modules among the construction module 410, the target emotion attribute node acquisition module 420, the video knowledge graph acquisition module 430, the node feature acquisition module 440, the description word probability distribution module 450, the description word determination module 460, and the target event description text acquisition module 470 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the construction module 410, the target emotion attribute node acquisition module 420, the video knowledge graph acquisition module 430, the node feature acquisition module 440, the description word probability distribution module 450, the description word determination module 460, and the target event description text acquisition module 470 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable method of integrating or packaging circuits, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the construction module 410, the target emotion attribute node acquisition module 420, the video knowledge graph acquisition module 430, the node feature acquisition module 440, the description word probability distribution module 450, the description word determination module 460, and the target event description text acquisition module 470 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0164] Figure 5 A block diagram of an electronic device suitable for implementing a method for generating event description text based on video data according to an embodiment of the present invention is shown.

[0165] like Figure 5 As shown, an electronic device according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 502 or programs loaded from a storage unit 508 into a random access memory (RAM) 503. Processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 501 may also include onboard memory for caching purposes. Processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0166] Various programs and data required for the operation of the electronic device are stored in RAM 503. The processor 501, ROM 502, and RAM 503 are connected to each other via a bus 504. The processor 501 executes the programs in ROM 502 and / or RAM 503 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. The processor 501 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0167] According to an embodiment of the present invention, the electronic device may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device may further include one or more of the following components connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage portion 508 including a hard disk; and a communication portion 509 including a network interface card such as a LAN card or a modem. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. Removable media 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 510 as needed, so that computer programs read from the removable media can be installed in the storage portion 508 as needed.

[0168] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0169] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above, and / or one or more memories other than ROM 502 and RAM 503.

[0170] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the method for generating event description text based on video data provided by an embodiment of the present invention.

[0171] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when the computer program is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0172] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 509, and / or installed from a removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0173] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509 and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-described functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0174] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0177] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for generating event description text based on video data, characterized in that: include: Constructing video scene graph data based on the video data to be processed, wherein the video scene graph data includes object nodes and edge relationships, wherein the object nodes represent human objects and object objects related to the human objects in the video data to be processed; Processing the to-be-processed video data based on the trained emotion classifier to obtain a target emotion attribute node, wherein the target emotion attribute node represents the emotion attribute of the character object; Update the video scene graph data based on the target emotion attribute node to obtain a video knowledge graph, wherein the video knowledge graph includes an edge relationship between the object node and the target emotion attribute node; Extracting features of target object nodes in the video knowledge graph to obtain node features; Determining the probability distribution of the i+1th description word according to the generated description word sequence, the target sentiment attribute node, and the node feature, wherein the generated description word sequence includes i description words, i≥0 and i is an integer; Using a decoding strategy, determining the description word corresponding to the maximum probability in the probability distribution of the i+1th description word as the i+1th description word; When the number of the generated description words is greater than or equal to a preset number threshold, a target event description text is obtained.

2. The method according to claim 1, characterized in that The trained emotion classifier is used to process the video data to obtain a target emotion attribute node, including: Processing the video data to be processed using a visual feature extraction layer to obtain visual features; Processing the video data to be processed using an audio feature extraction layer processor to obtain audio features; Processing the visual features based on a self-attention mechanism to obtain visual weighted features; Performing feature fusion on the visual weighted features and the audio features to obtain multimodal emotion features; The multimodal emotional features are classified into emotional attributes to obtain target emotional attribute nodes.

3. The method according to claim 1, characterized in that The trained sentiment classifier is determined based on the following operations: Obtain video training data and emotion category label data; Processing a multimodal emotion training feature using an activation function to obtain a predicted emotion probability distribution of an emotion category to which the video belongs, wherein the multimodal emotion training feature is obtained based on the video training data; Determining a loss function based on the predicted emotion probability distribution and the emotion category label data; The loss function is minimized according to the stochastic gradient descent algorithm to obtain a trained sentiment classifier.

4. The method according to claim 1, wherein The trained decoder is trained based on the following steps, Obtain training event description text, training object node and training emotion attribute node; Processing the training object nodes using a graph convolutional neural network to obtain training node context features; Processing the training node context features and event description text features according to a multi-layer perceptron to obtain a mean and a variance, wherein the event description text features are generated based on the training event description text; Processing the description word sequence, the training object node, and the training emotion attribute node generated before the tth time using a decoder to obtain a description word probability distribution of the training event description text at the tth time, where t is an integer greater than zero; Determining a variational lower bound based on the description word probability distribution, the mean, and the variance; The variational lower bound is maximized to obtain a trained decoder.

5. The method according to claim 4, characterized in that The method of processing the training object node using a graph convolutional neural network to obtain context features of the training node includes: According to the n-1th layer node representation matrix of the graph convolutional neural network, determine the nth layer node representation matrix; Processing the n-th layer training object node representation matrix and the n-th layer other node representation matrix according to the activation function to obtain the attention weight of the training object node to the other nodes, wherein the training object node corresponds to the n-th layer training object node representation matrix, and the other nodes correspond to the n-th layer other node representation matrix; The training node context feature is obtained according to the attention weight and the representation matrix of other nodes in the nth layer, where n is an integer greater than 1.

6. The method according to claim 4, characterized in that The method of using a decoder to process the description word sequence, the training object node, and the training emotion attribute node generated before the t-th moment to obtain the description word probability distribution of the training event description text at the t-th moment includes: Using a decoder to process the description word sequence generated before time t, the training emotion attribute node, and the training object node to obtain a decoded hidden layer representation feature at time t; The decoded hidden layer representation features at the t-th moment are processed using a normalization algorithm to obtain a probability distribution of description words of the training event description text at the t-th moment.

7. A device for generating event description text based on video data, characterized in that: include: A construction module is used to construct video scene graph data based on the video data to be processed, wherein the video scene graph data includes object nodes and edge relationships, and the object nodes represent human objects in the video data to be processed and object objects related to the human objects; A target emotion attribute node obtaining module is used to process the to-be-processed video data based on the trained emotion classifier to obtain a target emotion attribute node, wherein the target emotion attribute node represents the emotion attribute of the character object; A video knowledge graph obtaining module is used to update the video scene graph data based on the target emotion attribute node to obtain a video knowledge graph, wherein the video knowledge graph includes an edge relationship between the object node and the target emotion attribute node; A node feature acquisition module is used to extract features of target object nodes in the video knowledge graph to obtain node features; describe A word probability distribution module is used to determine the probability distribution of the i+1th description word based on the generated description word sequence, the target sentiment attribute node and the node feature, wherein the generated description word sequence includes i description words, i≥0 and i is an integer; a description word determination module, configured to determine the description word corresponding to the maximum probability in the probability distribution of the i+1th description word as the i+1th description word by using a decoding strategy; The target event description text obtaining module is used to obtain the target event description text when the number of the generated description words is greater than or equal to a preset number threshold.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images