Video description generation system based on relationship perception

Abstract

The application belongs to the technical field of cross-media generation, and particularly relates to a video description generation system based on relationship perception. The system comprises a video feature extraction network, a relationship perception visual feature coding network and a language decoding network. The video feature extraction network performs frame sampling on a video to obtain frame-level visual semantic features, target object sequence features and related information. The relationship perception visual feature coding network fuses the frame-level visual semantic features and reconstructively codes the target object sequence features in the video according to the relationship information. The language decoding network adopts an attention mechanism to fuse the feature information and generate a final description sentence. The application preextracts relationship information between significant objects in a video by using a scene graph analysis model, reconstructs object-level sequence features in the video by using a graph convolution, and fully utilizes spatial semantic and time sequence relationship information between objects in the video when generating a description sentence, so that the generation is more accurate, and the understanding ability of the model to visual information of the video is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of cross-media generation, and particularly relates to a video description generation system based on relationship perception. BACKGROUND

[0002] In recent years, the efficiency of information transmission has been improved with the development of science and technology, and videos have gradually replaced traditional media such as texts and voices to become the mainstream information acquisition channel for people. At the same time, the advent of the mobile Internet era has further lowered the threshold of video creation and dissemination, and the amount of video data has increased day by day. Therefore, how to enable computers to automatically understand videos has attracted widespread attention. The video description generation task (Video Captioning) aims to enable computers to generate a sentence or a paragraph of natural language describing the content of a given input video, and is one of the tasks that can most intuitively demonstrate the understanding ability of computers. Since the video description generation task needs to establish the connection between the features of different modalities, it is necessary to find a way to map the information of different modalities to the same vector space, which puts higher requirements on video feature extraction and feature modeling. With the continuous increase of video information, this task has very broad application scenarios, such as automatic commentary, navigation assistance, intelligent human-computer environment development, etc., and has great significance in promoting tourism, education and the development of computer science itself.

[0003] Existing video description generation methods can generally be divided into two categories, namely template-based description generation methods and deep learning-based description generation methods. The template-based method defines some special rules of natural language for the generation model in advance, such as subject, verb and object. Using such a syntax predefined template, the generation model can directly predict the words of each part of the sentence to generate a reasonable description for a given video, but the output is often similar and lacks diversity. At the same time, since the template-based method depends on predefined templates and language rules, careful rule design is required before training, which also causes the generated sentences to be highly rigid in form. With the continuous progress of deep learning, deep learning methods based on the "encoder-decoder" framework have been widely used in video description generation tasks since their introduction in the machine translation task. This kind of method generally uses a convolutional neural network (CNN) model to extract visual semantic features from a sequence of video frames, and encodes and fuses the features to represent the video. In the decoding stage, a recurrent neural network (RNN) model is used to generate the final description sentence according to the visual information transmitted by the encoder. In the visual information encoding stage, in addition to the conventional convolutional feature extraction, some more advanced methods will use object detection technology to pre-detect the objects in the video, and fuse the detected object information into the visual information representation to capture more fine-grained visual semantic information.

[0004] In summary, the existing video description generation methods have achieved good results and can generate reasonable description sentences for videos. However, there are still some problems that have not been well solved, such as subject errors in descriptions, inaccurate action prediction, and complex scenes with multiple objects that are difficult to understand. The main reason for these situations is that although the advanced methods for video description generation task model the object features in the video, they do not explicitly model the relationship between the objects in the video in a more fine-grained manner, resulting in the information in the video not being fully preserved. On the one hand, since the traditional feature extraction model does not explicitly model the relationship information, the model may lose some effective information, resulting in errors in the subject or verb of the generated description sentence. On the other hand, due to the limited size of the data set, the scene information involved often has a long tail effect, and some complex scenes may have fewer samples, so the model often ignores the global complex relationship information and only extracts part of the information commonly seen in the data set for prediction. The present invention is inspired by this, and a video description generation method containing three parts is designed to explicitly model the object-level feature relationships in the video, reconstruct the object semantic information in the video using graph convolution, and generate more quality descriptive sentences for the video. SUMMARY

[0005] The purpose of the present invention is to propose a relationship-aware video description generation method for videos to improve the quality of the generated language description.

[0006] The relationship-aware video description generation system provided by the present invention adopts a graph convolution network, including a video feature extraction network, a relationship-aware visual feature encoding network, and a language decoding network. The video feature extraction network samples the video to obtain a feature vector sequence of the sampled frames, performs scene graph analysis on the sampled frames to obtain the feature vector sequence of the salient objects and the relationship prediction information between any object pairs, and inputs them into the relationship-aware visual feature encoding network for feature reconstruction. The relationship-aware visual feature encoding network reconstructs and fuses the features from the frame and object levels, and outputs the feature representation of the entire video to the language decoding network. The language decoding network further fuses the feature information using an attention mechanism and samples to generate the final video description sentence. The present invention uses a scene graph analysis model to pre-extract the relationship information between the salient objects in the video, reconstructs the object-level sequence features in the video using graph convolution, and fully utilizes the spatial semantic relationship information and temporal relationship information between the objects in the video when generating the description sentence, making the generation more accurate.

[0007] In the application, the video feature extraction network comprises InceptionResnet-V2, I3D deep convolution network and scene graph analysis network ReIDNs; key frames of the video are sampled, the sampling is performed according to equal frame numbers, and finally 26 frames correspond to one video, each frame is mapped into a 2D feature with a dimension of 1,536 through the InceptionResnet-V2 network pre-trained on the ImageNet; 26 frame sets are extracted from the center of the 26 frames, and 3D features with a dimension of 1,024 of the video are obtained through the I3D network; meanwhile, scene graph features of each key frame on the video frame sequence are extracted through the ReIDNs network. The scene graph features comprise two parts, namely the features of the target objects in the frame and the relationship prediction information between the objects; finally, the features of one video comprise 2D features, 3D features and scene graph features.

[0008] In the application, the relationship-aware visual feature encoding network comprises a frame-level feature encoding network and a relationship-aware object feature enhancement network; the frame-level feature encoding network fuses the 2D features and the 3D features to capture the visual semantic features of the frame level of the video; the relationship-aware object feature enhancement network constructs a spatial and temporal relationship graph of the object sequence of the video according to the scene graph features, models and enhances the object sequence features through the graph network; wherein:

[0009] The frame-level feature encoding network fuses the frame-level visual semantic information through a bidirectional LSTM network, and the basic model is defined as follows:

[0010] V a,m =[V a ;V m ], (1)

[0011] V s =BiLSTM(V a,m ), (2)

[0012] wherein V a is the 2D feature extracted in the video; V m is the 3D feature; [;] represents splicing; V s is the frame-level feature representation vector after encoding.

[0013] The relationship-aware object feature enhancement network comprises spatial relationship awareness and temporal relationship awareness of the object sequence; first, the object features and related information (including object frames, scene graphs, etc.) of the first n confidence objects are extracted from each video frame in the video through ReIDNs. Then, the extracted objects are used as nodes to construct a graph structure with n nodes, and the information of each node in the graph is the feature vector of the object. For the edges of the graph, the adjacent matrix form representing the relationship information is used, that is If the ith object o i There exists a relationship pointing from the jth object o j Then there is In this way, a relation-aware spatial graph is constructed for all sampled frames to represent the semantic interaction information between objects.

[0014] Based on the spatial relation graph, the update process of the spatial relation-aware branch is defined as follows:

[0015]

[0016] where, denotes the sequence of n objects with d-dimensional features; V o,i denotes the feature vector of the ith object; is the transformation matrix; b is the bias; p represents the activation function (such as ReLU, Tanh, etc.); N(V o,i ) denotes the neighbor set of the ith node, i.e., the node set that has a weighted edge with the node (including the node itself); lab(V o,i , V o,j ) denotes the bias based on different relationship types between the ith object and the jth object, which is:

[0017]

[0018] The temporal relation-aware branch fuses the temporal relationship information of the object sequence into the object features. To capture the object relationship information in the temporal dimension, for the object in the tth frame only needs to model the relationship between it and the objects in the adjacent frames, i.e., to establish a relationship connection between adjacent frames. By calculating the Euclidean distance between the feature representation of the object and the feature representation of the objects in the adjacent t±1 frames , the similarity of the object to the objects in the adjacent frames can be evaluated. If the Euclidean distance between and is less than the threshold value λ, the pair of video objects can be considered as the same object appearing in two adjacent frames, and a connection with a value of 1 is established between them in the temporal relation graph. The relationship evaluation process can be expressed as:

[0019]

[0020]

[0021] Thus, an unweighted and undirected graph, i.e., the temporal relation graph ​On the basis of the constructed time sequence relation graph, the object feature vector of the video is updated by the graph convolution network, and the process is as follows:

[0022]

[0023] Wherein, W tem and b tem are learnable parameters; ρ represents an activation function; N (V o,i ) represents a neighbor set of the i th node.

[0024] Then, the updated object feature sequences of the two branches are fused by a multi-layer perceptron (MLP) layer, which can be represented as:

[0025] V′ o = ρ (W o [V o,R ; V o,T ] + b o ), (8)

[0026] Wherein, V o,R ; V o,T are updated spatial relation object feature sequences and time sequence relation object feature sequences respectively; is a learnable linear transformation matrix; b o is bias information; ρ is an activation function, generally a Tanh function. Thus, the video object feature representation V′ o based on relation perception information enhancement is obtained.

[0027] In the application, the language decoding network adopts a recurrent neural network as a generator to generate a sentence; a double-layer LSTM network with an attention mechanism is adopted as a decoder, and the double-layer LSTM network is an attention LSTM and a language decoding LSTM; the attention LSTM fuses global information at the current time, inputs an attention module to generate attention weights; the language decoding LSTM performs fine-grained decoding, and finally the output of the language decoding LSTM is used to predict a word at the next time.

[0028] The input of the attention LSTM is composed of four parts, which are the hidden state of the language LSTM at the last time, the word vector w t-1 of the predicted word at the last time, a frame-level feature vector output by a visual information encoder and an object feature sequence vector, and the output thereof is defined as follows:

[0029]

[0030]

[0031] where W e is the embedding matrix of word vectors; denotes the frame-level feature vector after global average; denotes the object feature sequence after global average; [;] denotes the concatenation operation of vectors; is the fused global information at the current time, which will be input to the attention network of the language decoder together with the frame-level feature V s and the object feature V′ o The calculation process of the attention network is as follows:

[0032]

[0033]

[0034] where, denotes the attention weight of the i-th frame feature at the t-th time step; denotes the attention weight of the i-th object feature at the t-th time step; W s , W o and U h are all learnable parameters. Through the above way, the attention weights of the frame-level feature and the object feature are calculated, and the new feature representation is obtained by weighting:

[0035]

[0036]

[0037]

[0038] where n and K are the number of sampled frames and the number of detected objects per frame, respectively; is a transformation matrix; b att is a bias; the updated feature of the attention module is obtained as and After that, further fusion is performed as shown in equation (15) to generate the output of the attention network

[0039] The language LSTM input is and the output is The process is as follows:

[0040]

[0041]

[0042] The language decoding network further adopts a two-layer MLP network as an information fusion mechanism to obtain a conditional probability distribution on a dictionary, which is defined as:

[0043]

[0044] The application provides a video description generation system based on relationship perception, and a work flow thereof is as follows:

[0045] (I) The video feature extraction network performs frame sampling on the video, uses a pre-trained deep CNN model to perform feature representation on the image, and extracts a frame-level visual feature vector sequence; uses an I3D network to obtain a motion feature representation in the time dimension of the video; uses a scene graph analysis model to extract a feature representation of an object in a frame and relationship prediction information, and obtains an object feature vector sequence at the target level.

[0046] (II) The relationship perception visual feature coding network is divided into a frame-level global feature coding branch and an object-level object feature coding branch; the frame-level global feature coding branch takes the frame-level visual feature vector sequence and the motion feature representation extracted by the I3D network as inputs to obtain a fused frame-level feature sequence; the object-level object feature coding branch takes the object feature vector sequence and the predicted relationship information as inputs, constructs a spatial relationship graph structure according to the relationship information, constructs a time sequence relationship graph structure according to the similarity of the feature vectors, and uses a graph convolution network to reconstruct the features to obtain an object-level feature sequence enhanced by the relationship information.

[0047] (III) The language decoding network takes the reconstructed frame-level feature sequence and object-level feature sequence as inputs, weights and fuses the two features through an attention mechanism, and outputs a sentence describing the video.

[0048] Compared with the existing related video description generation method, the method proposed by the application has two advantages, namely, fully utilizing the spatial relationship of specific target objects in the video and fully utilizing the time sequence relationship of specific target objects in the video. First, the utilization of the spatial relationship of specific target objects in the video is reflected in that the model of the method proposed by the application uses a pre-trained scene graph analysis model to predict relationship information between target objects, then constructs a directed and weighted spatial relationship graph, and uses a graph convolution network considering the relationship information for reconstruction. The previous method often ignores this part of information or uses an implicit modeling method for insufficient modeling. Second, the utilization of the time sequence relationship of specific target objects in the video is reflected in that the method proposed by the application uses the similarity between objects as a basis to construct a time sequence relationship adjacency matrix, and fully extracts and utilizes this information through a graph network update method.

[0049] In summary, the method for generating video description based on relationship perception of the present application makes full use of visual semantic information in the video, and the generated description sentence is more accurate. The present application can be applied to the generation of description for large-scale videos, and accurate description sentences corresponding to the video content are generated, thereby improving the usability and user experience. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The system diagram of the present application is shown.

[0051] Figure 2 The model framework in the present application is shown. DETAILED DESCRIPTION

[0052] As known from the background, the existing video description generation method does not make full use of the relationship information of the target objects in the video. The present application studies the problem, introduces scene graph analysis and graph convolution network technology, reconstructs the visual information in the video, fully considers the semantic association information between the target objects in the video, including spatial relationship and time sequence information, and uses two-layer LSTM as a decoder to generate the final description sentence. The model of the present application is suitable for all video description generation technologies based on the "encoder-decoder" mode, and significantly improves the accuracy of the generated sentence.

[0053] As shown in Figure 1 The video description generation system based on relationship perception of the present application includes a video feature extraction network, a relationship perception visual feature encoding network, and a language decoding network. The process is as follows:

[0054] One: the visual feature extraction network, frame sampling is carried out for the obtained video, and a visual feature vector representation 26*1,536 is obtained by using a deep convolutional network. The deep convolutional network can select a deep convolutional model in an image classification task, and the InceptionResnet-V2 network is adopted in the application. In order to capture dynamic information in the frame sequence, the I3D model pre-trained on the Kinetics dataset extracts 1,024-dimensional 3D features of the video. Further, in order to obtain fine-grained information of the target object in the video, the scene graph analysis model ReIDNs pre-trained on the Visual Genome dataset is used to extract features of the video frame sequence. The model will detect all possible objects, i.e. undetermined objects, on the frame. By calculating the joint intersection (Intersection of Union, IoU) between each undetermined object, some redundant objects will be removed. Then, the remaining undetermined objects are sorted according to the confidence value, and the top 20 confidence value objects are taken from each frame as input objects; at the same time, the relationship between the selected 20 objects is also predicted. The Visual Genome dataset contains 51 kinds of relationships, including action words, position relationship words, unknown or no relationship, etc. A special mark self is further added to represent the self-correlation relationship, i.e. the relationship of the object itself, to construct a relationship list containing 52 elements. That is, the relationship information a spa ∈[0,52].

[0055] In one embodiment, two levels of extraction are performed for video feature extraction, one is the whole frame, and the other is the target object information inside the frame. The above information is input into the relationship-aware visual feature coding network for further reconstruction and fusion.

[0056] Two: the relationship-aware visual feature coding network reconstructs the obtained visual features, including a frame-level feature coding network and a relationship-aware object feature enhancement network. The specific process of reconstruction is as follows:

[0057] (1) The frame-level feature coding network inputs the features extracted by the 2D and 3D networks after splicing, and encodes the frame-level features by using a bidirectional LSTM.

[0058] (2) A spatial relationship graph and a time sequence relationship graph are constructed. A spatial relationship graph is constructed for the object sequence in each video frame, and the object features are updated using the spatial relationship graph. Each object is regarded as a node in the graph. n objects with d-dimensional features are represented as The space relationship graph corresponds to an adjacency matrix, and the edges are the relationship information predicted by the ReIDNs. At the same time, a time sequence relationship graph is constructed for the entire video frame sequence, the similarity information between the objects in the current frame and the objects in the adjacent frame is calculated, the weight of the graph is obtained, and then an adjacency matrix in the time sequence dimension is constructed The similarity evaluation process is as follows:

[0059]

[0060]

[0061] (Three), the spatial relationship graph and the time sequence relationship graph are reconstructed, respectively obtaining the object feature sequence enhanced by the spatial relationship and the object feature sequence enhanced by the time sequence relationship, and the two features are fused through a multi-layer perceptron (MLP) layer.

[0062] In process two, the video information is processed in two levels, one is to fuse the frame sequence features, and a bidirectional LSTM is used to fully extract the information in the frame sequence; the other is to reconstruct the target object sequence, in which both the fine-grained visual semantic relationship of the target in the frame and the time sequence relationship of the target in the similar position between frames are considered. The feature reconstruction process uses a graph convolution network considering relationship information, so that the reconstructed feature is more suitable for video sentence description generation.

[0063] Three: the language decoding network adopts a two-layer LSTM with an attention mechanism as the decoder. The first layer is used as coarse-grained decoding, and the input includes the hidden state of the previous time language LSTM The word vector w of the word predicted in the last time t-1 The frame-level feature vector and the object feature sequence vector output by the visual information encoder, and the output is The input of the attention network is The frame-level feature sequence V output in process two s and the reconstructed object feature sequence V o ′ The calculation process is as follows:

[0064]

[0065]

[0066]

[0067]

[0068]

[0069] The second layer LSTM input comprises the output of the first layer and the output of the attention network The output of the second layer is used to predict the next time word probability distribution p, and the specific operation is as follows:

[0070]

[0071]

[0072]

[0073] The specific process visualization is shown in Figure 2 , and the video description sentence can be obtained by sequentially performing steps one to three. In addition, the BeamSearch method is used to select the optimal sentence when generating the sentence description, which effectively avoids the sampling deviation of the word. In the method proposed in the application, the value of BeamSize is set to 3 to balance the relationship between accuracy and time cost.

[0074] The method example proposed in the application selects the data set MSVD and MSR-VTT. MSVD contains 1,970 videos, and MSVD contains 10,000 videos. Both of them use the standard data set division, and the BLEU-4, CIDEr, METEOR, and ROUGE-L scores on the MSVD test set are 55.2, 100.1, 36.5, and 74.1 respectively. On the MSR-VTT, the BLEU-4, Cider, Meteor, and Rouge scores are 42.5, 51.4, 28.5, and 62.4 respectively, and the performance is greatly improved compared with the previous method.

[0075] The application is not only suitable for the model framework, but also suitable for all video description generation methods based on encoding-decoding, and is also suitable for video description generation methods that do not use recurrent neural networks as decoders. The feature reconstruction process of the application is added to the encoding stage without changing the decoder scheme.

[0076] Although the application has been disclosed as above with a preferred embodiment, it is not intended to limit the application, and any person skilled in the art can make possible changes and modifications to the technical solutions of the application by using the disclosed technical content without departing from the spirit and scope of the application. Therefore, any simple modification, equivalent change and modification of the above embodiments according to the technical essence of the application, which does not deviate from the technical solutions of the application, belongs to the protection scope of the technical solutions of the application.

Claims

1. A relation-aware based video description generation system, characterized in that, Adopt the graph convolution network, including video feature extraction network, relationship perception visual feature coding network, language decoding network, wherein, the video feature extraction network carries out sampling to the video, obtains the feature vector sequence of the sampling frame, carries out scene graph analysis to the sampling frame, obtains the feature vector sequence of the salient target and the relationship prediction information between any target pair, and is input to the relationship perception visual feature coding network and carries out feature reconstruction, the relationship perception visual feature coding network reconstructs the feature from frame and target two levels, fuses, and the feature representation of whole video is output, and is input to language decoding network, the language decoding network adopts attention mechanism and further fuses feature information, samples and generates the final video description sentence, the work flow is: (I) the video feature extraction network carries out frame sampling to the video, utilizes the pre-trained deep CNN model to carry out feature representation to the image, extracts the frame level visual feature vector sequence, utilizes I3D network to obtain the motion feature representation on the time dimension of the video, utilizes scene graph analysis model to extract the feature representation of the object in the frame and the relationship prediction information, obtains the object feature vector sequence of target level, (II) the relationship perception visual feature coding network is divided into two branches, the global feature coding branch of frame level and the object feature coding branch of target level, the global feature coding branch of frame level takes the frame level visual feature vector sequence and the motion feature representation extracted by I3D network as input, obtains the fused frame level feature sequence, the object feature coding branch of target level takes the object feature vector sequence and the predicted relationship information as input, constructs the spatial relationship graph structure according to the relationship information, constructs the time sequence relationship graph structure according to the feature vector similarity, adopts the graph convolution network to carry out feature reconstruction, obtains the object level feature sequence enhanced by relationship information, (III) the language decoding network takes the reconstructed frame level feature sequence, object level feature sequence as input, fuses two kinds of features by attention mechanism weighting, and outputs the sentence describing the video.

2. The video description generation system of claim 1, wherein, The video feature extraction network contains InceptionResnet-V2, I3D deep convolution network and scene graph analysis network ReIDNs, carries out key frame sampling to the video, sampling is according to equal frame number sampling, a video corresponds to 26 frames finally, each frame is mapped to 1,536 dimensions 2D features through the InceptionResnet-V2 network pre-trained on ImageNet, 26 frames are extracted as the center 26 fragment sets, 1,024 dimensions 3D features of the video are obtained through I3D network, at the same time, the scene graph features of each key frame on the video frame sequence are extracted through ReIDNs network, the scene graph features include the features extracted by the target objects in the frame and the relationship prediction information between each other, finally, for a video, Its features contain 2D features, 3D features and scene graph features.

3. The video description generation system of claim 2, wherein, The relationship-aware visual feature encoding network comprises a frame-level feature encoding network and a relationship-aware object feature enhancement network; the frame-level feature encoding network fuses 2D features and 3D features to capture visual semantic features at the frame level; and the relationship-aware object feature enhancement network constructs a spatial and temporal relationship graph of a video object sequence according to scene graph features, models and enhances object sequence features in the form of a graph network; wherein: The frame-level feature encoding network fuses frame-level visual semantic information through a bidirectional LSTM network, and the basic model is defined as follows: V a,m = [V a ; V m ] (1) V s = BiLSTM(V a,m ) (2) where V a is the 2D feature extracted in the video; V m is the 3D feature; [; ] denotes concatenation; V s is the encoded frame-level feature representation vector; The relationship-aware object feature enhancement network includes two branches of spatial relationship awareness and temporal relationship awareness for object sequences; first, for each video frame extracted in the video, object features and related information of the first n confidence objects are extracted by ReIDNs; then, using these extracted objects as nodes, a graph structure with n nodes is constructed, and the information of each node in the graph is the feature vector of the object; for the edges of the graph, the adjacent matrix form representing the relationship information is used, that is If the i-th object o i There is a relationship j between the j-th object o Then there is In this way, a relationship-aware spatial graph is constructed for all sampled frames to represent the semantic interaction information between objects; On the basis of the spatial relationship graph, the updating process of the spatial relationship-aware branch on the feature sequence is defined as follows: where, denotes a sequence of n objects with d-dimensional features; V o,i denotes the feature vector of the i-th object; is a transformation matrix; b is a bias; p denotes an activation function; N(V o,i ) denotes the neighbor set of the i-th node, i.e., the set of nodes that have a weighted edge with the node; lab(V o,i ,V o,j denotes the bias based on different relationship types between the i-th object and the j-th object, which has: The temporal relationship aware branch fuses the temporal relationship information of the object sequence into the object features; in order to capture the object relationship information in the temporal dimension, for the object in the t-th frame , the relationship between the object and the objects in the adjacent frames is modeled, i.e. the relationship connection between the adjacent frames is established; by calculating the Euclidean distance between the feature representation of the object and the feature representation of the objects in the adjacent t±1 frames , the similarity of the object to the objects in the adjacent frames can be evaluated; if and the Euclidean distance between them is less than a threshold λ, i.e. the pair of video objects are regarded as the same object appearing in two adjacent frames, the existence edge is marked in the temporal relationship graph, and the connection between the two objects is established with a value of 1; the relationship evaluation process is expressed as: Thus, a directed graph without weight is constructed, which is the time sequence relationship graph On the basis of the constructed time sequence relationship graph, the object feature vector of the video is updated by the graph convolution network, and the process is as follows: where W tem with b tem are learnable parameters; p denotes an activation function; N(V o,i ) denotes the neighborhood set of the i-th node. Then, the object feature sequences updated by the two branches are fused through a multi-layer perceptron (MLP) layer, and are represented as follows: V o ′ = p(W o [V o,R ;V o,T ]+ b o ) (8) wherein V o,R ; V o,T are the updated spatial and temporal relationship object feature sequences, respectively; is a learnable linear transformation matrix; b o is bias information; and ρ is an activation function; thus, the video object feature representation V o ′ is obtained based on relationship-aware information enhancement.

4. The video description generation system of claim 3, wherein, The language decoding network adopts a recurrent neural network as a generator to generate a sentence; adopts a double-layer LSTM network with an attention mechanism as a decoder, and the double-layer LSTM network comprises an attention LSTM and a language decoding LSTM; the attention LSTM fuses global information at a current time, inputs an attention module to generate attention weights; the language decoding LSTM performs fine-grained decoding, and finally outputs a word at a next time through the language decoding LSTM; The input of the attention LSTM consists of four parts, which are the hidden state of the language LSTM at the last time step the word vector w of the predicted word at the last time step t-1 the frame-level feature vector and the object feature sequence vector output by the visual information encoder, and the output is defined as follows: where W e is the embedding matrix of word vectors; represents the frame-level feature vector after global average; represents the object feature sequence after global average; [;] represents the concatenation operation of vectors; is the fused global information at the current moment, which will be combined with the frame-level feature V s and the object feature V o ′ together input into the attention network of the language decoder; the calculation process of the attention network is as follows: wherein, denotes the attention weight of the i-th frame feature at the t-th time step; denotes the attention weight of the i-th object feature at the t-th time step; W s , W o and U h are learnable parameters; in this way, the attention weights for the frame-level features and the object features are calculated; the weighted new feature representation is denoted as: Wherein, n and K are the number of frames and the number of objects detected per frame, respectively; is a transformation matrix; b att is a bias; the feature updated by the attention module and After that, further fusion is performed as shown in equation (15) to generate the output of the attention network The language LSTM input is The output is The process is as follows: The language decoding network adopts a two-layer MLP network as an information fusion mechanism to obtain a conditional probability distribution on a dictionary, and is defined as follows:

Images