Event description text generation method, device and equipment based on video data

Abstract

The invention provides an event description text generation method, device and equipment based on video data, which can be applied to the field of computer vision. The method comprises the steps that video scene graph data is constructed based on to-be-processed video data, the video scene graph data comprises object nodes and edge relations, and the object nodes represent character objects in the to-be-processed video data and object objects related to the character objects; processing the to-be-processed video data based on the trained emotion classifier to obtain a target emotion attribute node, the target emotion attribute node representing the emotion attribute of the character object; updating the video scene graph data based on the target emotional attribute node to obtain a video knowledge graph, the video knowledge graph comprising an edge relationship between the object node and the target emotional attribute node; and processing the video knowledge graph by using the trained decoder to obtain a target event description text for describing the behavior of the character object.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, device and equipment for generating event description text based on video data. Background Art

[0002] With the popularization of video surveillance systems, intelligent video event analysis and description technology has become a research hotspot in academia and industry. Traditional methods mainly focus on the semantic content recognition of events, that is, detecting objects, behaviors and their relationships in the video, while ignoring the emotional information contained in the event. However, emotional factors play an indispensable role in understanding the overall picture of the event and portraying the nature of the event. For example, crowd gathering events accompanied by anger often mean potential group conflict risks; while crowds in a celebratory atmosphere usually have no safety hazards. Therefore, mining emotional clues in video events is of great significance for comprehensively controlling the development of events and responding to them in a timely manner. Summary of the invention

[0003] In view of the above problems, the present invention provides a method, device and apparatus for generating event description text based on video data.

[0004] According to a first aspect of the present invention, a method for generating event description text based on video data is provided, comprising: constructing video scene graph data based on the video data to be processed, the video scene graph data comprising object nodes and edge relationships, the object nodes representing the human objects and object objects related to the human objects in the video data to be processed; processing the video data to be processed based on a trained emotion classifier to obtain a target emotion attribute node, the target emotion attribute node representing the emotion attribute of the human object; updating the video scene graph data based on the target emotion attribute node to obtain a video knowledge graph, the video knowledge graph comprising edge relationships between the object nodes and the target emotion attribute nodes; extracting features of the target object nodes in the video knowledge graph to obtain node features; determining the i+1th description word probability distribution according to a generated description word sequence, the target emotion attribute node and the node features, the generated description word sequence comprising i description words, i≥0 and i is an integer; using a decoding strategy, determining the description word corresponding to the maximum probability in the i+1th description word probability distribution as the i+1th description word; and obtaining the target event description text when the number of generated description words is greater than or equal to a preset number threshold.

[0005] According to an embodiment of the present invention, the video data to be processed is processed based on the trained emotion classifier to obtain a target emotion attribute node, including: using a visual feature extraction layer to process the video data to be processed to obtain visual features; using an audio feature extraction layer processor to process the video data to be processed to obtain audio features; processing the visual features based on a self-attention mechanism to obtain visual weighted features; performing feature fusion on the visual weighted features and the audio features to obtain multimodal emotion features; performing emotion attribute classification on the multimodal emotion features to obtain a target emotion attribute node.

[0006] According to an embodiment of the present invention, the trained emotion classifier is determined based on the following operations: obtaining video training data and emotion category label data; using an activation function to process multimodal emotion training features to obtain a predicted emotion probability distribution of the emotion category to which the video belongs, wherein the multimodal emotion training features are obtained based on the video training data; determining a loss function based on the predicted emotion probability distribution and the emotion category label data; and minimizing the loss function based on a stochastic gradient descent algorithm to obtain a trained emotion classifier.

[0007] According to an embodiment of the present invention, the trained decoder is trained based on the following steps: obtaining training event description text, training object nodes and training emotion attribute nodes; using a graph convolutional neural network to process the training object nodes to obtain training node context features; processing the training node context features and event description text features according to a multi-layer perceptron to obtain a mean and variance, and the event description text features are generated based on the training event description text; using a decoder to process the description word sequence, training object nodes and training emotion attribute nodes generated before the tth moment to obtain the description word probability distribution of the training event description text at the tth moment, where t is an integer greater than zero; determining a variational lower bound based on the description word probability distribution, mean and variance; maximizing the variational lower bound to obtain a trained decoder.

[0008] According to an embodiment of the present invention, a graph convolutional neural network is used to process training object nodes to obtain contextual features of the training nodes, including: determining an n-th layer node representation matrix based on an n-1-th layer node representation matrix of the graph convolutional neural network; processing an n-th layer training object node representation matrix and an n-th layer other node representation matrix based on an activation function to obtain attention weights of the training object nodes to other nodes, wherein the training object nodes correspond to the n-th layer training object node representation matrix, and the other nodes correspond to the n-th layer other node representation matrix; obtaining contextual features of the training nodes based on the attention weights and the n-th layer other node representation matrix, where n is an integer greater than 1.

[0009] According to an embodiment of the present invention, a decoder is used to process a description word sequence, a training object node, and a training sentiment attribute node generated before the tth moment to obtain a description word probability distribution of a training event description text at the tth moment, including: using a decoder to process a description word sequence, a training sentiment attribute node, and a training object node generated before the tth moment to obtain a decoded hidden layer representation feature at the tth moment; using a normalization algorithm to process the decoded hidden layer representation feature at the tth moment to obtain a description word probability distribution of the training event description text at the tth moment.

[0010] The second aspect of the present invention provides an event description text generation device based on video data, including: a construction module, which is used to construct video scene graph data based on the video data to be processed, the video scene graph data includes object nodes and edge relationships, and the object nodes represent the character objects and object objects related to the character objects in the video data to be processed; a target emotion attribute node acquisition module, which is used to process the video data to be processed based on the trained emotion classifier to obtain the target emotion attribute node, and the target emotion attribute node represents the emotion attribute of the character object; a video knowledge graph acquisition module, which is used to update the video scene graph data based on the target emotion attribute node to obtain the video knowledge graph, and the video knowledge graph includes the object nodes and the target emotion attribute node. The edge relationship between the emotion attribute nodes; the node feature acquisition module is used to extract the features of the target object nodes in the video knowledge graph to obtain the node features; the description word probability distribution module is used to determine the i+1th description word probability distribution according to the generated description word sequence, the target emotion attribute node and the node features, the generated description word sequence includes i description words, i≥0 and i is an integer; the description word determination module is used to use the decoding strategy to determine the description word corresponding to the maximum probability in the i+1th description word probability distribution as the i+1th description word; the target event description text acquisition module is used to obtain the target event description text when the number of generated description words is greater than or equal to the preset number threshold.

[0011] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0012] The fourth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0013] According to an embodiment of the present invention, by constructing a video scene graph based on the video data to be processed, the video data to be processed is processed based on the trained emotion classifier to obtain the target emotion attribute node. The addition of the target emotion attribute node makes the understanding of video surveillance no longer limited to the visual level, but also captures the emotional atmosphere in the scene, thereby more comprehensively reflecting the nature of the event. The video scene graph data is updated based on the target emotion attribute node to obtain a video knowledge graph, which can better understand the dynamic events in the video. This dynamic understanding ability can adapt to complex and changeable monitoring scenes, and can timely detect abnormal emotions or potential risks. The trained decoder is used to process the video knowledge graph to obtain a target event description text for describing the behavior of the character object. The target event description text carries the target emotion attribute. By introducing the target emotion attribute node, the decoded text can not only accurately depict the content of the event, but also reflect the subtle emotional color, and better judge the risk level and emergency status of the event. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0015] Figure 1 An application scenario diagram showing a method and device for generating event description text based on video data according to an embodiment of the present invention;

[0016] Figure 2 A flowchart of a method for generating event description text based on video data according to an embodiment of the present invention is shown;

[0017] Figure 3 A schematic diagram showing a method for generating event description text based on video data according to an embodiment of the present invention is shown;

[0018] Figure 4 It shows a structural block diagram of a device for generating event description text based on video data according to an embodiment of the present invention;

[0019] Figure 5 A block diagram of an electronic device suitable for implementing a method for generating event description text based on video data according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0020] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0021] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0022] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0023] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0024] Traditional methods mainly focus on the semantic content recognition of events, that is, detecting objects, behaviors and their relationships in the video, while ignoring the emotional information contained in the event. However, emotional factors play an indispensable role in understanding the overall picture of the event and portraying the nature of the event. For example, crowd gathering events accompanied by anger often mean potential group conflict risks; while crowds in a festive atmosphere usually have no safety hazards. Therefore, mining emotional clues in video events is of great significance for comprehensively controlling the development of events and responding to them in a timely manner. However, most of the current research focuses on event type recognition, while ignoring the value of emotional semantics for the task of generating event descriptions. The above problems restrict the development of emotion-driven intelligent video event analysis technology.

[0025] In view of this, the present invention provides a method for generating event description text based on video data, and an apparatus and device for generating event description text based on video data. The method comprises: constructing video scene graph data based on the video data to be processed, the video scene graph data comprising object nodes and edge relationships, the object nodes representing the human objects and object objects related to the human objects in the video data to be processed; processing the video data to be processed based on the trained emotion classifier to obtain the target emotion attribute node, the target emotion attribute node representing the emotion attribute of the human object; updating the video scene graph data based on the target emotion attribute node to obtain the video knowledge graph, the video knowledge graph comprising the edge relationship between the object node and the target emotion attribute node; processing the video knowledge graph using the trained decoder to obtain the target event description text for describing the behavior of the human object.

[0026] It should be noted that the event description text generation method based on video data and the event description text generation device based on video data provided by the present invention can be used in the field of computer vision, and can also be used in any field other than the field of computer vision, such as the field of urban transportation. Therefore, the application field of the event description text generation method based on video data and the event description text generation device based on video data provided by the present invention is not limited.

[0027] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present invention provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0029] Figure 1 An application scenario diagram is shown of a method and device for generating event description text based on video data according to an embodiment of the present invention.

[0030] like Figure 1 As shown, the application scenario according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0031] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0034] It should be noted that the event description text generation method based on video data provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the event description text generation device based on video data provided in the embodiment of the present invention can generally be set in the server 105. The event description text generation method based on video data provided in the embodiment of the present invention can also be performed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the event description text generation device based on video data provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0036] Figure 2 A flow chart of a method for generating event description text based on video data according to an embodiment of the present invention is shown.

[0037] like Figure 2 As shown, the event description text generation method based on video data of this embodiment includes operations S210 to S270, and the event description text generation method based on video data can be executed by an electronic device.

[0038] In operation S210, video scene graph data is constructed based on the video data to be processed, where the video scene graph data includes object nodes and edge relationships.

[0039] In operation S220, the video data to be processed is processed based on the trained emotion classifier to obtain a target emotion attribute node, where the target emotion attribute node represents the emotion attribute of the person object.

[0040] In operation S230, the video scene graph data is updated based on the target emotion attribute node to obtain a video knowledge graph, which includes edge relationships between object nodes and target emotion attribute nodes.

[0041] In operation S240, feature extraction is performed on the target object node in the video knowledge graph to obtain node features.

[0042] In operation S250, the probability distribution of the i+1th description word is determined according to the generated description word sequence, the target sentiment attribute node and the node features, and the generated description word sequence includes i description words, i≥0 and i is an integer.

[0043] In operation S260, a decoding strategy is used to determine the description word corresponding to the maximum probability in the probability distribution of the i+1th description word as the i+1th description word.

[0044] In operation S270, when the number of the generated description words is greater than or equal to a preset number threshold, a target event description text is obtained.

[0045] According to an embodiment of the present invention, video scene graph data can be constructed based on the video data to be processed, and the video scene graph data may include object nodes and edge relationships, and the object nodes represent the person objects and object objects related to the person objects in the video data to be processed. For example, there are "person", "football", and "kick" in the video scene graph, where "person" and "football" can be represented as object nodes, and "kick" can be represented as an edge relationship.

[0046] According to an embodiment of the present invention, an existing video scene graph data generation algorithm (such as RelTR, etc.) can be used to construct a scene graph for each segment of video data to be processed. ,in , Represents a collection of object nodes in the scene. Indicates object nodes, Represents the total number of objects in the video to be processed. Represents the set of relationship edges between object nodes. Describing Object Nodes and object nodes The type of relationship between .

[0047] According to an embodiment of the present invention, the target emotion attribute node can be obtained by processing the video data to be processed based on the trained emotion classifier, and the target emotion attribute node represents the emotion attribute of the character object. The emotion attribute can include emotional states such as happiness, sadness, anger, fear, etc., but is not limited to this, and the embodiment of the present invention does not limit this.

[0048] According to an embodiment of the present invention, in order to integrate the overall emotional information of the video data to be processed, in the constructed scene graph Introducing sentiment attribute nodes .remember for The video knowledge graph includes the edge relationship between the object node and the target emotion attribute node. The video knowledge graph can be obtained by updating the video scene graph data based on the target emotion attribute node. Finally, the emotion-enhanced video knowledge graph is obtained. ,in Associating edges with newly added sentiment attribute nodes, reflecting Dependencies with other nodes.

[0049] According to an embodiment of the present invention, by updating the video scene graph data based on the target emotion attribute node, a video knowledge graph can be obtained, and the video knowledge graph can include edge relationships between object nodes and target emotion attribute nodes. Introducing emotion attribute nodes on the basis of traditional video scene graphs enhances the depiction of the emotional semantics of events by graph representation. The emotion-enhanced scene event graph integrates rich audiovisual clues, emotional information, and temporal associations between events, providing a powerful knowledge representation for a deep understanding of video events. At the same time, the introduction of emotion nodes as latent variables of the generative model enables the decoded text to not only accurately depict the content of the event, but also reflect subtle emotional colors. This emotional and factual description plays an important role in helping managers fully grasp the status of video events and make timely and effective decision responses.

[0050] According to an embodiment of the present invention, a target event description text for describing the behavior of a character object can be obtained by processing a video knowledge graph using a trained decoder. The decoder can use a decoder based on a recurrent neural network (RNN), a decoder based on a Transformer, a decoder based on a large language model, etc., but is not limited to this, and the embodiment of the present invention does not limit this. The video knowledge graph may include multiple characters and different emotional attribute nodes. For example, in the initial video scene graph, there are "people", "football", and "kick", where "people" and "football" can be represented as object nodes, and "kick" can be represented as an edge relationship. The video data is processed based on the trained emotion classifier to obtain a target emotional attribute node with a "happy" attribute, and the initial video scene graph data is updated based on the target emotional attribute node to obtain a video knowledge graph. The video knowledge graph is processed using a trained decoder to obtain a target event description text of "people playing football happily".

[0051] According to an embodiment of the present invention, by constructing a video scene graph based on the video data to be processed, the video data to be processed is processed based on the trained emotion classifier to obtain the target emotion attribute node. The addition of the target emotion attribute node makes the understanding of video surveillance no longer limited to the visual level, but also captures the emotional atmosphere in the scene, thereby more comprehensively reflecting the nature of the event. The video scene graph data is updated based on the target emotion attribute node to obtain a video knowledge graph, which can better understand the dynamic events in the video. This dynamic understanding ability can adapt to complex and changeable monitoring scenes, and can timely discover abnormal emotions or potential risks. The trained decoder is used to process the video knowledge graph to obtain a target event description text for describing the behavior of the character object. The target event description text has the target emotion attribute. By introducing the target emotion attribute node, the decoded text can not only accurately depict the event content, but also reflect the subtle emotional color, better judge the risk level and emergency state of the event, make up for the shortcomings of single modal analysis and the shortcomings of ignoring emotional factors, and provide new ideas for tasks such as smart security and auxiliary decision-making.

[0052] According to an embodiment of the present invention, the video data to be processed is processed based on the trained emotion classifier to obtain a target emotion attribute node, including: using a visual feature extraction layer to process the video data to be processed to obtain visual features; using an audio feature extraction layer processor to process the video data to be processed to obtain audio features; processing the visual features based on a self-attention mechanism to obtain visual weighted features; performing feature fusion on the visual weighted features and the audio features to obtain multimodal emotion features; performing emotion attribute classification on the multimodal emotion features to obtain a target emotion attribute node.

[0053] According to an embodiment of the present invention, visual features can be obtained by processing the video data to be processed using a visual feature extraction layer. The visual feature extraction layer can use a visual transformer (ViT), an efficient neural network (EfficientNet), etc. At the same time, the audio feature extraction layer processor can be used to process the video data to be processed to obtain audio features. The audio feature extraction layer can use a VGG-like Audio Feature Extractor (VGGish). In this embodiment, the EfficientNet-B5 Model (EfficientNet-B5) can be selected as the visual feature extraction layer to output a 2048-dimensional visual feature vector; the VGGish network extracts a 128-dimensional audio feature vector.

[0054] According to an embodiment of the present invention, visual features are processed based on the self-attention mechanism to obtain visual weighted features, as specifically shown in formula (1).

[0055] (1);

[0056] Among them, Q represents the query matrix of visual features, Key represents the key matrix of visual features, and V represents the value matrix of visual features. Represents the visual weighted feature, and i represents the i-th video in the video data to be processed.

[0057] According to an embodiment of the present invention, frames with higher relevance can obtain greater weights, and a multi-head attention structure can be adopted. The number of heads can be set to 8, but is not limited to this. The embodiment of the present invention does not limit the number of heads.

[0058] According to an embodiment of the present invention, by fusing visual weighted features and audio features, multimodal emotion features can be obtained, as specifically shown in formula (2).

[0059] (2);

[0060] in, represents the multimodal emotional features represented by the i-th video, represents the vector concatenation operation, represents a multi-layer perceptron network, represents the visual weighted features of the i-th video in the video data to be processed, Represents the audio features corresponding to the i-th video.

[0061] According to an embodiment of the present invention, by classifying the emotion attributes of the multimodal emotion features, a target emotion attribute node can be obtained, as specifically shown in formula (3).

[0062] (3);

[0063] in, represents the target emotional attribute node corresponding to the i-th video, W is the emotional attribute node embedding matrix, Represents the multimodal emotional features represented by the i-th video.

[0064] According to an embodiment of the present invention, by combining visual features and audio features, emotional information in a video can be captured more comprehensively. A single modality (such as only audio or only vision) may not be able to fully express emotions, while multimodal fusion can make up for the shortcomings of a single modality and improve the accuracy of emotion classification.

[0065] According to an embodiment of the present invention, a trained decoder is used to process a video knowledge graph to obtain a target event description text for describing the behavior of a character object, including: extracting features of the target object nodes in the video knowledge graph to obtain node features; determining the i+1th description word probability distribution based on a generated description word sequence, target emotional attribute nodes and node features, the generated description word sequence includes i description words, i≥0 and i is an integer; using a decoding strategy, determining the description word corresponding to the maximum probability in the i+1th description word probability distribution as the i+1th description word; and obtaining the target event description text when the number of generated description words is greater than or equal to a preset number threshold.

[0066] According to an embodiment of the present invention, feature extraction is performed on target object nodes in a video knowledge graph to obtain node features.

[0067] According to an embodiment of the present invention, based on the generated description word sequence, target sentiment attribute node and node features, the probability distribution of the i+1th description word can be determined, and the generated description word sequence can include i description words, i≥0 and i is an integer. The generated description word sequence, target sentiment attribute node and node features can be processed by a Transformer decoder to determine the probability distribution of the i+1th description word, as shown in formula (4).

[0068] (4);

[0069] in, Indicates time The previously generated description word sequence, x represents the node feature, z represents the target sentiment attribute node, represents the probability distribution of the i+1th description word, y represents the target event description text, Represents the parameters involved in the loss function, and T represents a total of T moments.

[0070] According to an embodiment of the present invention, the decoder can be used to determine the description word corresponding to the maximum probability in the probability distribution of the i+1th description word as the i+1th description word, as shown in formula (5).

[0071] (5);

[0072] in, Represents the descriptive word with the highest probability in the target event description text, represents the event description text sequence generated at time t. It represents the description word generated at the predicted t+1th time, that is, the i+1th description word mentioned above, x represents the node feature, and z represents the target sentiment attribute node.

[0073] According to an embodiment of the present invention, when the number of generated description words is greater than or equal to a preset number threshold, a target event description text can be obtained. , which indicates a sequence starting with the start symbol. Repeat the following steps until terminator to get the target event description text.

[0074] According to an embodiment of the present invention, a decoder can be used to describe an event, and the features corresponding to the emotion attribute nodes rich in emotion can be embedded into the decoding process. By introducing the emotion attribute, the system can be helped to better judge the risk level and emergency state of the event. Involving emotion as a latent variable in the decoding process can ensure that the generated event description text not only conforms to the actual content of the event but also reflects the corresponding emotional color.

[0075] According to an embodiment of the present invention, the trained emotion classifier is determined based on the following operations: obtaining video training data and emotion category label data; using an activation function to process multimodal emotion training features to obtain a predicted emotion probability distribution of the emotion category to which the video belongs, wherein the multimodal emotion training features are obtained based on the video training data; determining a loss function based on the predicted emotion probability distribution and the emotion category label data; and minimizing the loss function based on a stochastic gradient descent algorithm to obtain a trained emotion classifier.

[0076] According to an embodiment of the present invention, video training data and emotion category label data can be obtained. Video training data covering different scenes and event types can be collected from multiple channels such as video websites and monitoring systems. In order to enrich the emotional diversity of the data, various emotional states such as happiness, sadness, anger, fear, disgust, and surprise are included as much as possible. In this embodiment, a total of about 10,000 video clips can be collected, and the length of each clip is between 10 seconds and 5 minutes.

[0077] According to an embodiment of the present invention, multiple professionals can watch each video, subjectively judge the emotion category and intensity of the video according to the video content, and make corresponding annotations, and finally obtain emotion category label data.

[0078] According to an embodiment of the present invention, the multimodal emotion training feature is obtained based on the video training data. By processing the multimodal emotion training feature using an activation function, the predicted emotion probability distribution of the emotion category to which the video belongs can be obtained, as specifically shown in formula (6).

[0079] (6);

[0080] in, represents the predicted emotional probability distribution of the emotional category of the i-th video, Softmax represents the activation function, The embedding matrix representing the total number of sentiment categories C, are the classifier parameters, Represents the multimodal emotional features represented by the i-th video.

[0081] According to an embodiment of the present invention, a loss function may be determined based on the predicted emotion probability distribution and emotion category label data, as specifically shown in formula (7).

[0082] (7);

[0083] in, represents the loss function, is the total number of preset emotion categories, N is the total number of videos, represents the predicted emotional probability distribution of the i-th video belonging to emotional category j, Indicates the emotional category label of the i-th video Dimension value.

[0084] According to an embodiment of the present invention, the parameters involved in the loss function can be recorded as .

[0085] According to an embodiment of the present invention, the loss function is processed according to the stochastic gradient descent algorithm, the learning rate can be set to 0.0001, the number of training iteration cycles can be 100, and the batch size can be 32. Finally, a trained sentiment classifier can be obtained, as shown in formula (8).

[0086] (8);

[0087] in, represents the trained sentiment classifier, Represents the parameters involved in minimizing the loss function , Represents the loss function.

[0088] According to an embodiment of the present invention, by training with multimodal emotion features and emotion annotation data, an efficient and accurate video emotion classifier is established, which provides key emotion recognition capabilities for subsequent event analysis. The emotion classifier can automatically process a large amount of video data, quickly output emotion attribute nodes, and save time and cost for manual annotation. According to an embodiment of the present invention, the trained decoder is trained based on the following steps: obtaining training event description text, training object nodes, and training emotion attribute nodes; processing training object nodes using a graph convolutional neural network to obtain training node context features; processing training node context features and event description text features according to a multi-layer perceptron to obtain mean and variance, and event description text features are generated based on training event description text; using a decoder to process description word sequences, training object nodes, and training emotion attribute nodes generated before the tth moment, to obtain description word probability distribution of the training event description text at the tth moment, t is an integer greater than zero; determining a variational lower bound based on the description word probability distribution, mean, and variance; maximizing the variational lower bound to obtain a trained decoder.

[0089] According to an embodiment of the present invention, the decoder may be composed of two parts, an inference network and a generation network. A training event description text, a training object node, and a training emotion attribute node may be obtained. The training object node is processed using a graph convolutional neural network to obtain training node context features.

[0090] According to an embodiment of the present invention, the event description text features are generated based on the training event description text. According to the multilayer perceptron (MLP) network processing training node context features and event description text features, the mean and variance can be obtained, as shown in formula (9):

[0091] (9);

[0092] in, represents a multi-layer perceptron network, represents the context feature of the training node, Indicates that there are training nodes related to the emotional attribute node e in the i-th video. indivual, Indicates the text features of the event description. represents the mean, Represents variance.

[0093] According to an embodiment of the present invention, by using a decoder to process the description word sequence, training object nodes and training emotion attribute nodes generated before the tth moment, the description word probability distribution of the training event description text at the tth moment can be obtained, where t is an integer greater than zero.

[0094] According to an embodiment of the present invention, the variational lower bound is determined based on the probability distribution, mean and variance of the description words, as shown in formulas (10), (11), (12) and (13).

[0095] (10);

[0096] Among them, ELBO represents the variational lower bound, represents the parameters involved in minimizing the loss function in the generative network, Represents the reconstruction loss, that is, Under the sampled target sentiment attribute node z, the generative network reconstructs the log-likelihood of the target event description text y. The second term is the posterior distribution With prior distribution The information divergence (Kullback-Leibler Divergence, KL) between them is used to constrain the learned posterior to be as close to the prior as possible, Represents the parameters involved in the inference network.

[0097] In practice, the reconstruction error term can be rewritten as:

[0098] (11);

[0099] Among them, L represents the number of samples of sentiment attribute nodes.

[0100] According to an embodiment of the present invention, the prior distribution It can be assumed to be a standard Gaussian distribution ,and The KL divergence between can be obtained analytically:

[0101] (12);

[0102] in, It can represent the mean of the i-th video. It can represent the variance of the i-th video.

[0103] In summary, the variational lower bound (10) can be approximately expressed as:

[0104] (13);

[0105] Among them, ELBO represents the variational lower bound, It can represent the mean of the i-th video. It can represent the variance of the i-th video.

[0106] According to an embodiment of the present invention, the variational lower bound is maximized to obtain a trained decoder.

[0107] According to an embodiment of the present invention, during model training, an adaptive moment estimation (Adaptive Moment Estimation, Adam) optimizer can be used to maximize the variational lower bound (Evidence Lower Bound, ELBO), the learning rate can be set to 0.0005, the number of training rounds can be 150, and the batch size can be 48. The constructed video dataset contains 1000 video samples, which are divided into a training set, a validation set, and a test set according to 8:1:1. Finally, the content quality of the generated text can be evaluated using the automated recall-oriented summary evaluation index (Recall-Oriented Understudy for Gisting Evaluation, ROUGE) and the explicit ordering machine translation evaluation index (Metric for Evaluation of Translation with Explicit Ordering, METEOR) indicators. Experimental results show that the corresponding system of the present invention has achieved the recall-oriented summary evaluation index-longest common subsequence (Recall-Oriented Understudy for Gisting Evaluation-Longest Common Subsequence, ROUGE-L) and METEOR values ​​of 35.2 and 27.5 respectively on the video dataset. Qualitative analysis also confirms that the event description text generated by the method of the present invention better depicts the emotional semantics of the video event, and the description is richer and more vivid.

[0108] According to an embodiment of the present invention, a graph convolutional neural network is used to process training object nodes to obtain contextual features of the training nodes, including: determining an n-th layer node representation matrix based on an n-1-th layer node representation matrix of the graph convolutional neural network; processing an n-th layer training object node representation matrix and an n-th layer other node representation matrix based on an activation function to obtain attention weights of the training object nodes to other nodes, wherein the training object nodes correspond to the n-th layer training object node representation matrix, and the other nodes correspond to the n-th layer other node representation matrix; obtaining contextual features of the training nodes based on the attention weights and the n-th layer other node representation matrix, where n is an integer greater than 1.

[0109] According to an embodiment of the present invention, the node representation matrix of the nth layer can be determined based on the node representation matrix of the n-1th layer of the graph convolutional neural network, as specifically shown in formula (14).

[0110] (14);

[0111] in, is the n-th layer node representation matrix, represents the n-1th layer node representation matrix, is the initial representation matrix of the node, represents the correlation function, A matrix representing the adjacency matrix of the video knowledge graph plus the self-connection matrix, for The degree matrix of are trainable weights.

[0112] According to an embodiment of the present invention, in addition to information aggregation at the node level, the interaction relationship between nodes is also required, especially to capture the dependency between the emotion node and the object node. The training object node corresponds to the n-th layer training object node representation matrix, and the other nodes correspond to the n-th layer other node representation matrix. The n-th layer training object node representation matrix and the n-th layer other node representation matrix can be processed according to the activation function to obtain the attention weight of the training object node to other nodes, as shown in formula (15):

[0113] (15);

[0114] Among them, Softmax represents the activation function, Indicates the training object node K to other nodes The attention weight, represents the training object node of the nth layer, K represents the matrix, Indicates other nodes in the nth layer represents the matrix, and is the parameter matrix of the attention network.

[0115] According to an embodiment of the present invention, the training node context feature can be obtained based on the attention weight and the representation matrix of other nodes in the nth layer, as specifically shown in formula (16).

[0116] (16);

[0117] in, represents the context feature of the training node, represents the attention weight, Indicates other nodes in the nth layer represents the matrix, Represents the parameter matrix of the attention network.

[0118] According to an embodiment of the present invention, by using graph neural network modeling, the causal logic and emotional transfer pattern between events are mined. The graph neural network is used to realize information interaction between events, in which each node not only represents a person object or an object object, but also carries emotional information.

[0119] According to an embodiment of the present invention, a decoder is used to process a description word sequence, a training object node, and a training sentiment attribute node generated before the tth moment to obtain a description word probability distribution of a training event description text at the tth moment, including: using a decoder to process a description word sequence, a training sentiment attribute node, and a training object node generated before the tth moment to obtain a decoded hidden layer representation feature at the tth moment; using a normalization algorithm to process the decoded hidden layer representation feature at the tth moment to obtain a description word probability distribution of the training event description text at the tth moment.

[0120] According to an embodiment of the present invention, a decoder is used to process the description word sequence, training emotion attribute nodes and training object nodes generated before the tth moment to obtain the decoded hidden layer representation features at the tth moment, as specifically shown in formula (17).

[0121] (17)

[0122] in, represents the decoded hidden layer representation feature at time t, represents the parameters involved in minimizing the loss function in the generative network, Indicates time The previously generated description word sequence, To train sentiment attribute nodes, Represents the representation matrix of the L-th layer training object node K, Indicates that there are training nodes related to the emotional attribute node e in the i-th video. indivual.

[0123] According to an embodiment of the present invention, a normalization algorithm is used to process the decoded hidden layer representation features at the t-th moment to obtain the description word probability distribution of the training event description text at the t-th moment, as specifically shown in formula (18).

[0124] (18);

[0125] in, represents the probability distribution of the description words of the training event description text at time t, represents the decoding hidden layer representation feature at the tth moment, Softmax represents the activation function, and Represents the parameter matrix in the decoder.

[0126] Figure 3 A schematic diagram of a method for generating event description text based on video data according to an embodiment of the present invention is shown.

[0127] like Figure 3 As shown, the event description text generation method based on video data includes three steps, namely, emotion attribute node generation 310, video knowledge graph construction 320 and event description text generation 330. Specifically, emotion attribute node generation 310 includes: obtaining the video surveillance data to be processed, inputting the video data to be processed into the audio feature extractor, and outputting the audio features, and inputting the video data to be processed into the visual feature extractor, and outputting the visual features. According to the visual features and audio features, multimodal emotion features can be obtained. The multimodal emotion features are input into the emotion classifier, and the emotion attribute nodes can be output. The video knowledge graph construction 320 includes: video scene graph data can be constructed based on the video data to be processed. The emotion attribute nodes and video scene graph data output by the emotion attribute node generation 310 are updated using a graph neural network to obtain video scene graph data. The event description text generation 330 includes: using a decoder to process the target emotion attribute node and the video knowledge graph to obtain the target event description text.

[0128] Figure 4 The structure block diagram of the device for generating event description text based on video data according to an embodiment of the present invention is shown.

[0129] like Figure 4 As shown, the event description text generation device based on video data of this embodiment includes a construction module 410, a target emotion attribute node acquisition module 420, a video knowledge graph acquisition module 430, a node feature acquisition module 440, a description word probability distribution module 450, a description word determination module 460 and a target event description text acquisition module 470.

[0130] The construction module 410 is used to process the video data to be processed based on the trained emotion classifier to obtain a target emotion attribute node, where the target emotion attribute node represents the emotion attribute of the person object.

[0131] The target sentiment attribute node obtaining module 420 is used to process each traffic sequence data and topology map using a graph neural network to obtain traffic spatiotemporal characteristics.

[0132] The video knowledge graph obtaining module 430 is used to update the video scene graph data based on the target emotion attribute node to obtain the video knowledge graph, and the video knowledge graph includes the edge relationship between the object node and the target emotion attribute node.

[0133] The node feature obtaining module 440 is used to extract features of the target object nodes in the video knowledge graph to obtain node features.

[0134] The description word probability distribution module 450 is used to determine the i+1th description word probability distribution according to the generated description word sequence, target sentiment attribute node and node features, and the generated description word sequence includes i description words, i≥0 and i is an integer.

[0135] The description word determination module 460 is used to determine the description word corresponding to the maximum probability in the probability distribution of the i+1th description word as the i+1th description word by using a decoding strategy.

[0136] The target event description text obtaining module 470 is used to obtain the target event description text when the number of generated description words is greater than or equal to a preset number threshold.

[0137] According to an embodiment of the present invention, by constructing a video scene graph based on the video data to be processed, the video data to be processed is processed based on the trained emotion classifier to obtain the target emotion attribute node. The addition of the target emotion attribute node makes the understanding of video surveillance no longer limited to the visual level, but also captures the emotional atmosphere in the scene, thereby more comprehensively reflecting the nature of the event. The video scene graph data is updated based on the target emotion attribute node to obtain a video knowledge graph, which can better understand the dynamic events in the video. This dynamic understanding ability can adapt to complex and changeable monitoring scenes, and can timely discover abnormal emotions or potential risks. The trained decoder is used to process the video knowledge graph to obtain a target event description text for describing the behavior of the character object. The target event description text has a target emotion attribute. By introducing the target emotion attribute node, the decoded text can not only accurately depict the content of the event, but also reflect the subtle emotional color, and better judge the risk level and emergency state of the event.

[0138] According to an embodiment of the present invention, the target emotion attribute node obtaining module 420 includes: a visual feature obtaining unit, an audio feature obtaining unit, a visual weighted feature obtaining unit, a feature fusion unit and an emotion attribute classification unit.

[0139] The visual feature obtaining unit is used to process the video data to be processed using the visual feature extraction layer to obtain visual features.

[0140] The audio feature obtaining unit is used to process the video data to be processed using the audio feature extraction layer processor to obtain audio features.

[0141] The visual weighted feature acquisition unit is used to process visual features based on the self-attention mechanism to obtain visual weighted features.

[0142] The feature fusion unit is used to fuse the visual weighted features and the audio features to obtain multimodal emotional features.

[0143] The sentiment attribute classification unit is used to classify the sentiment attributes of the multimodal sentiment features to obtain the target sentiment attribute node.

[0144] According to an embodiment of the present invention, the target emotion attribute node obtaining module 420 includes: a first obtaining unit, a predicted emotion probability distribution unit, a loss function obtaining unit and a first training unit.

[0145] The first acquisition unit is used to acquire video training data and emotion category label data.

[0146] The predicted emotion probability distribution unit is used to process the multimodal emotion training features using an activation function to obtain the predicted emotion probability distribution of the emotion category to which the video belongs. The multimodal emotion training features are obtained based on video training data.

[0147] The loss function obtaining unit is used to process the predicted emotion probability distribution and emotion category label data using the cross entropy loss function to obtain the loss function.

[0148] The first training unit is used to process the loss function according to the stochastic gradient descent algorithm to obtain a trained sentiment classifier.

[0149] According to an embodiment of the present invention, the video knowledge graph obtaining module 430 includes: a second acquisition unit, a training node context feature obtaining unit, a mean variance obtaining unit, a description word probability distribution unit, a variational lower bound determination unit and a second training unit.

[0150] The second acquisition unit is used to acquire the training event description text, the training object node and the training emotion attribute node.

[0151] The training node context feature obtaining unit is used to process the training object nodes using a graph convolutional neural network to obtain training node context features.

[0152] The mean-variance obtaining unit is used to process the training node context features and event description text features according to the multi-layer perceptron to obtain the mean and variance. The event description text features are generated based on the training event description text.

[0153] The description word probability distribution unit is used to use the decoder to process the description word sequence, training object nodes and training sentiment attribute nodes generated before the tth moment to obtain the description word probability distribution of the training event description text at the tth moment, where t is an integer greater than zero.

[0154] The variational lower bound determination unit is used to determine the variational lower bound according to the probability distribution, mean and variance of the description words.

[0155] The second training unit is used to maximize the variational lower bound to obtain a trained decoder.

[0156] According to an embodiment of the present invention, the training node context feature obtaining unit includes: a node representation matrix determination subunit, an attention weight obtaining subunit and a training node context feature obtaining subunit.

[0157] The node representation matrix determination subunit is used to determine the n-th layer node representation matrix according to the n-1-th layer node representation matrix of the graph convolutional neural network.

[0158] The attention weight obtaining subunit is used to process the n-th layer training object node representation matrix and the n-th layer other node representation matrix according to the activation function to obtain the attention weight of the training object node to other nodes, wherein the training object node corresponds to the n-th layer training object node representation matrix, and the other nodes correspond to the n-th layer other node representation matrix.

[0159] The training node context feature obtaining subunit is used to obtain the training node context feature according to the attention weight and the representation matrix of other nodes in the nth layer.

[0160] According to an embodiment of the present invention, the description word probability distribution unit includes: a decoding hidden layer representation feature subunit and a description word probability distribution subunit.

[0161] The decoding hidden layer representation feature subunit is used to use the decoder to process the description word sequence, training sentiment attribute nodes and training object nodes generated before the tth moment to obtain the decoding hidden layer representation feature at the tth moment.

[0162] The description word probability distribution subunit is used to process the decoded hidden layer representation features at the t-th moment using a normalization algorithm to obtain the description word probability distribution of the training event description text at the t-th moment.

[0163] According to an embodiment of the present invention, any multiple modules among the construction module 410, the target emotion attribute node acquisition module 420, the video knowledge graph acquisition module 430, the node feature acquisition module 440, the description word probability distribution module 450, the description word determination module 460 and the target event description text acquisition module 470 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the construction module 410, the target emotion attribute node acquisition module 420, the video knowledge graph acquisition module 430, the node feature acquisition module 440, the description word probability distribution module 450, the description word determination module 460 and the target event description text acquisition module 470 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by an appropriate combination of any of them. Alternatively, at least one of the construction module 410, the target emotion attribute node acquisition module 420, the video knowledge graph acquisition module 430, the node feature acquisition module 440, the description word probability distribution module 450, the description word determination module 460 and the target event description text acquisition module 470 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0164] Figure 5 A block diagram of an electronic device suitable for implementing a method for generating event description text based on video data according to an embodiment of the present invention is shown.

[0165] like Figure 5 As shown, the electronic device according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0166] In RAM 503, various programs and data required for the operation of the electronic device are stored. The processor 501, ROM 502 and RAM 503 are connected to each other via a bus 504. The processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. The processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0167] According to an embodiment of the present invention, the electronic device may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0168] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0169] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0170] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the event description text generation method based on video data provided by the embodiment of the present invention.

[0171] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0172] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0173] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0174] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0175] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0177] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for generating event description text based on video data, characterized in that: include: Constructing video scene graph data based on the video data to be processed, wherein the video scene graph data includes object nodes and edge relationships, wherein the object nodes represent character objects and object objects related to the character objects in the video data to be processed; Processing the to-be-processed video data based on the trained emotion classifier to obtain a target emotion attribute node, wherein the target emotion attribute node represents the emotion attribute of the character object; Based on the target emotion attribute node, the video scene graph data is updated to obtain a video knowledge graph, wherein the video knowledge graph includes an edge relationship between the object node and the target emotion attribute node; Extracting features of target object nodes in the video knowledge graph to obtain node features; Determine the probability distribution of the i+1th description word according to the generated description word sequence, the target sentiment attribute node and the node feature, wherein the generated description word sequence includes i description words, i≥0 and i is an integer; Using a decoding strategy, determining the description word corresponding to the maximum probability in the probability distribution of the i+1th description word as the i+1th description word; When the number of the generated description words is greater than or equal to a preset number threshold, a target event description text is obtained.

2. The method according to claim 1, characterized in that: The step of processing the video data to be processed based on the trained emotion classifier to obtain a target emotion attribute node includes: Processing the video data to be processed by using a visual feature extraction layer to obtain visual features; Processing the video data to be processed by an audio feature extraction layer processor to obtain audio features; Processing the visual features based on a self-attention mechanism to obtain visual weighted features; Performing feature fusion on the visual weighted features and the audio features to obtain multimodal emotional features; The multimodal emotional features are classified into emotional attributes to obtain target emotional attribute nodes.

3. The method according to claim 1, characterized in that The trained sentiment classifier is determined based on the following operations: Obtain video training data and emotion category label data; Processing the multimodal emotion training features using an activation function to obtain a predicted emotion probability distribution of the emotion category to which the video belongs, wherein the multimodal emotion training features are obtained based on the video training data; Determining a loss function based on the predicted emotion probability distribution and emotion category label data; The loss function is minimized according to the stochastic gradient descent algorithm to obtain a trained sentiment classifier.

4. The method according to claim 1, characterized in that: The trained decoder is trained based on the following steps, Obtain training event description text, training object node and training sentiment attribute node; Processing the training object nodes using a graph convolutional neural network to obtain training node context features; Processing the training node context features and event description text features according to a multi-layer perceptron to obtain a mean and a variance, wherein the event description text features are generated based on the training event description text; Processing the description word sequence, the training object node and the training sentiment attribute node generated before the tth time by using a decoder to obtain the description word probability distribution of the training event description text at the tth time, where t is an integer greater than zero; Determining a variational lower bound according to the probability distribution of the description word, the mean, and the variance; The variational lower bound is maximized to obtain a trained decoder.

5. The method according to claim 4, characterized in that The step of processing the training object node using a graph convolutional neural network to obtain a training node context feature includes: According to the node representation matrix of the n-1th layer of the graph convolutional neural network, determine the node representation matrix of the nth layer; Processing the n-th layer training object node representation matrix and the n-th layer other node representation matrix according to the activation function to obtain the attention weight of the training object node to the other nodes, wherein the training object node corresponds to the n-th layer training object node representation matrix, and the other nodes correspond to the n-th layer other node representation matrix; The training node context feature is obtained according to the attention weight and the representation matrix of other nodes in the nth layer, where n is an integer greater than 1.

6. The method according to claim 4, characterized in that The method of using a decoder to process the description word sequence, the training object node, and the training emotion attribute node generated before the tth time to obtain the description word probability distribution of the training event description text at the tth time includes: Using a decoder to process the description word sequence generated before the tth moment, the training emotion attribute node and the training object node, to obtain the decoded hidden layer representation feature at the tth moment; The decoded hidden layer representation features at the t-th moment are processed using a normalization algorithm to obtain the probability distribution of the description words of the training event description text at the t-th moment.

7. A device for generating event description text based on video data, characterized in that: include: A construction module, configured to construct video scene graph data based on the video data to be processed, wherein the video scene graph data includes object nodes and edge relationships, wherein the object nodes represent character objects and object objects related to the character objects in the video data to be processed; A target emotion attribute node obtaining module is used to process the to-be-processed video data based on the trained emotion classifier to obtain a target emotion attribute node, wherein the target emotion attribute node represents the emotion attribute of the character object; A video knowledge graph obtaining module, used to update the video scene graph data based on the target emotion attribute node to obtain a video knowledge graph, wherein the video knowledge graph includes an edge relationship between the object node and the target emotion attribute node; A node feature acquisition module is used to extract features of target object nodes in the video knowledge graph to obtain node features; describe A word probability distribution module, used to determine the i+1th description word probability distribution according to the generated description word sequence, the target sentiment attribute node and the node feature, wherein the generated description word sequence includes i description words, i≥0 and i is an integer; A description word determination module, used to determine the description word corresponding to the maximum probability in the probability distribution of the i+1th description word as the i+1th description word by using a decoding strategy; The target event description text obtaining module is used to obtain the target event description text when the number of the generated description words is greater than or equal to a preset number threshold.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images