A video description method based on semantic disambiguation structured coding

Abstract

The application belongs to the field of computer vision, and discloses a video description method based on semantic disambiguation structured coding. The application proposes to introduce prior knowledge such as a knowledge graph to construct the relationship between objects in a video (a concept semantic graph), so as to obtain a structured coding of a deeper level of understanding of the semantic relationship of the video. On the basis of the concept semantic graph, according to the guidance of the semantic of the video scene, a relationship most conforming to the current video context is dynamically learned from various relationships of the same pair of objects to eliminate the semantic ambiguity problem existing between the objects, so as to achieve semantic disambiguation structured coding. A cross-domain guidance relationship learning strategy is proposed, which analyzes each object and its relationship in a description sentence to fit the learning of the concept semantic graph in the model, so as to better learn each object and the relationship between the objects in the video. The method of the application can realize more accurate and comprehensive video description.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, and particularly relates to a video description method based on semantic disambiguation structured coding. BACKGROUND

[0002] The video description task is to automatically convert the information in the video into a natural language sentence that can describe the video content. It has a wide application prospect in many aspects of modern life, work, study and the like, such as video retrieval, security detection and assistance for the visually impaired group, etc. In the coding process of video description, in order to generate a description sentence that can better reflect the content of the video, it means that the details in the video need to be coded. However, most of the existing video description content coding methods mainly focus on convolution at the frame level to extract features, which undoubtedly ignores the information such as objects in the frame and their semantic relationships, thereby causing the details in the video to be unable to be coded.

[0003] Video description is the reasoning and generalization of objects and their relationships in the video, and is an advanced semantic analysis task. The convolution coding method only from the frame level has two obvious deficiencies: (1) the feature extractor used for frame feature extraction directly convolves the frame as the smallest unit to extract features, which ignores the semantic information such as objects in the frame and their relationships, so that the model cannot understand the image in depth; (2) such feature extractor is usually pre-trained in a recognition task with class information as the label, which leads to the features extracted by the feature extractor are often low-level semantic information, which has a huge "semantic gap" with the high-level semantic features of natural language.

[0004] How to obtain more accurate and comprehensive video description is a challenging work. SUMMARY

[0005] The purpose of the present application is to overcome at least one deficiency of the prior art and provide a video description method based on semantic disambiguation structured coding.

[0006] The inventors have found that using prior knowledge such as knowledge graph to guide the model to learn the relationship of objects in the video can achieve cross-modal semantic mapping relationship from vision to text, which can undoubtedly code more details in the video.

[0007] The technical scheme adopted by the present application is:

[0008] A video description method based on semantic disambiguation structured coding, comprising the following steps:

[0009] S1) Construct a concept semantic graph:

[0010] The set of visual concepts and the relationship weight between the visual concepts are screened out by using the knowledge graph, and a sub-knowledge graph meeting the set is segmented;

[0011] The sub-knowledge graph is instantiated, i.e., a concept semantic graph is constructed for each frame in the video sample, and a concept semantic graph sequence of each video is obtained;

[0012] S2) Constructing a concept semantic graph based on semantic disambiguation:

[0013] On the basis of the concept semantic graph, the relationship between the objects of semantic disambiguation is learned by the scene semantic information of the video, so that the model dynamically learns the relationship meeting the current context from multiple relationships of each pair of objects, thereby obtaining a concept semantic graph of semantic disambiguation;

[0014] S3) Graph convolution network learning based on the concept semantic graph:

[0015] The concept semantic graph sequence is subjected to learning of the time sequence evolution process by the graph convolution network based on the concept semantic graph, so as to realize learning and reasoning of the concept semantic graph, and obtain a structured coding sequence of the video frame

[0016] S4) Cross-domain guided relationship learning:

[0017] The artificially annotated description sentence is parsed into a sentence scene graph expressing objects and their relationships, and the learning of the concept semantic graph is guided and constrained by the sentence scene graph;

[0018] S5) Language model decoding:

[0019] The video frame feature sequence is concatenated with the structured coding feature sequence to serve as a visual coding feature sequence of the video to supplement the global and action information of the frame;

[0020] The visual and audio features are respectively input into two Transformer models, and the output features obtained are concatenated to generate a description sentence.

[0021] In some instances of the video description method, in step S1:

[0022] Each object probability score of each frame in the video frame sequence V = {F1, F2, …, F T} is detected by a target detection method, and a probability score matrix S = O x T is obtained to represent the dynamic changes of the objects in the video, wherein S n represents the score of each object in the object set O in the nth frame;

[0023] All relationship weights between the same pair of objects are accumulated as the weight of the pair of objects to form an edge matrix AKG ∈R N×N , i.e. if any two nodes <i,j> have a direct edge in the graph, then is the accumulated sum of the corresponding edge values w ij , otherwise 0, to obtain the concept semantic graph sequence of each video.

[0024] In some examples of the video description method, in step S2:

[0025] with other objects except the selected object pair <i,j> as the scene semantics, and using an attention mechanism to learn the weight of the selected object pair <i,j> in different relationships ,

[0026] multiply the learned weight value of the object pair <i,j> in the kth relationship corresponding relationship aggregated into a relationship vector as the new weight value of the object <i,j> wherein, graph weight; w * : word vector of relationship r * .

[0027] to realize multi-relationship dynamic learning with the updated edge matrix to obtain the concept semantic graph with semantic disambiguation.

[0028] In some examples of the video description method, in step S3:

[0029] using a score fusion network to fuse the video score matrix S, wherein, α n,m represents the influence of the mth frame score S m on the nth frame score S n .

[0030] multiply the fused score matrix with the corresponding object word vector as the graph object node feature sequence of the video wherein, represents the fused score of each object in the nth frame after fusion, represents the graph object node feature matrix in the nth frame represents the word vector of the object set O.

[0031] perform structured information propagation on the concept semantic graph by a graph convolution network model to obtain Aggregating the updated node features of the network into structured encoding features of frames Finally obtaining a structured encoding sequence of video frames

[0032] The functions f(·), g(·), and h(·) are 1x1 convolutional networks.

[0033] In some examples of the video description method, in step S4:

[0034] Parsing the annotated description sentence of the video i into a sentence scene graph SG i and encoding the sentence scene graph SG using a scene graph encoding model to obtain an encoded vector i

[0035] Fusing the concept semantic graph sequence learned by the graph neural network into the output vector of the video

[0036] Projecting the encoded vector and the output vector of the video i to the same space to obtain a transposed sentence scene graph vector and a transposed video concept semantic graph fusion vector and calculating the similarity of and as part of the loss function to guide and constrain the learning of the concept semantic graph, W1 and W2 are learnable matrices.

[0037] In some examples of the video description method, in steps S4 and / or S5, the loss function optimization step includes:

[0038] The description sentence loss adopts a cross-entropy loss, and the calculation formula is as follows:

[0039]

[0040] where w 1:t-1 is the word sequence generated before time step t; θ is a learnable matrix; w 1:T is the annotated sentence;

[0041] The relationship learning loss, the calculation formula is as follows:

[0042]

[0043] where dist() is the cosine distance between the description sentence scene graph encoding vector and the corresponding video structured encoding feature, i is the i-th video; SG is the description sentence scene graph; and KG is the video structured encoding.​

[0044] Model total loss loss total Composed of the above two parts:

[0045] loss total = λ1loss sent + λ2loss rel

[0046] Where λ 1-2 is a hyperparameter of the proportion of the two losses.

[0047] In some examples of video description methods, in step S5:

[0048] The video frame feature sequence is concatenated with the structured coding feature sequence to form the visual coding feature sequence of the video to supplement the global and action information of the frame;

[0049] Considering that the video is multi-modal data, the audio feature sequence is also added.

[0050] The visual features and audio features are input into two Transformer models respectively, and the obtained visual output features V d and audio output features A d are concatenated to generate a description sentence.

[0051] V d = Transformer V (V e )

[0052] A d = Transformer A (A e )

[0053] sent = {w1, w2, …, w l} = softmax(FC(concat(V d , A d )))

[0054] Where Transformer V is the visual encoder Transformer, Transformer A is the audio encoder Transformer, concat is feature concatenation, and FC is a fully connected layer.

[0055] In some examples of video description methods, word vector features are extracted by a TransE model.

[0056] In some examples of the video description method, the audio features are extracted by a VGGish model.

[0057] In some examples of the video description method, the visual features are extracted by an I3D model.

[0058] In some examples of the video description method, the parameters of the graph convolution network model based on the concept semantic graph are set as follows:

[0059] The dimensions of the 3 graph convolution layers are 100, 256 and 512 respectively;

[0060] The audio and visual modes of the Transformer are 128 and 1024 respectively, each using one layer, and H=4 is used in the multi-head attention block;

[0061] The model uses an Adam optimizer, β=(0.9, 0.99), a smoothing parameter γ=0.7, and a batch size of 16;

[0062] In order to regularize the weights of the model, the model sets Dropout=0.1 and acts on the output of the position encoding, before the residual of each sub-layer and after the first layer inside the multi-modal generator.

[0063] The above features can be combined arbitrarily without conflict.

[0064] The beneficial effects of the present application are:

[0065] The present application proposes to introduce prior knowledge such as a knowledge graph to construct the relationship between objects in the video (concept semantic graph) to obtain a structured encoding of the deeper semantic relationship of the video; on the basis of the concept semantic graph, according to the guidance of the video scene semantics, dynamically learn the relationship that best fits the current video context from the multiple relationships of the same pair of objects to eliminate the semantic ambiguity problem between the objects, so as to achieve a structured encoding of semantic disambiguation; a cross-domain guided relationship learning strategy is proposed, which fits the learning of the concept semantic graph in the model by analyzing the objects and their relationships in the description sentence, so as to better learn the relationships between the objects in the video. The method of the present application can realize more accurate and comprehensive video description. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is the model framework diagram of the technical scheme of the present application.

[0067] Figure 2 is the graph convolution network framework based on the concept semantic graph of the present application.

[0068] Figure 3 is the effectiveness schematic diagram of the semantic disambiguation concept semantic graph module of the present application.

[0069] Figure 4 is a schematic diagram of the effectiveness of the cross-domain guided learning module of the present application.

[0070] Figure 5 is a description result display of the method of the present application on the ActivityNet Captions data set. DETAILED DESCRIPTION

[0071] In order to have a clearer understanding of the technical features, objectives and effects of the present application, the embodiments of the present application are described below in conjunction with the accompanying drawings, but the present application is not limited to the specific embodiments described below, which are illustrative rather than limiting, and those of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, which are all within the protection of the present application.

[0072] A video description method based on semantic disambiguation structured coding, comprising the following steps:

[0073] S1) constructing a concept semantic graph:

[0074] Filtering out the set of visual concepts and the relationship weight between visual concepts using a knowledge graph, and segmenting out a sub-knowledge graph that meets the set;

[0075] Instantiating the sub-knowledge graph, i.e., constructing a concept semantic graph for each frame in the video sample, to obtain a concept semantic graph sequence for each video;

[0076] S2) constructing a concept semantic graph based on semantic disambiguation:

[0077] On the basis of the concept semantic graph, the relationship between the objects of semantic disambiguation is learned by the scene semantic information of the video to guide the model, so that the model dynamically learns the relationship that meets the current context from a plurality of relationships of each pair of objects, thereby obtaining a concept semantic graph of semantic disambiguation;

[0078] S3) graph convolution network learning based on the concept semantic graph:

[0079] The concept semantic graph sequence is subjected to learning of the time sequence evolution process by the graph convolution network based on the concept semantic graph, to realize learning and reasoning of the concept semantic graph, and obtain a structured coding sequence of the video frame

[0080] S4) cross-domain guided relationship learning:

[0081] The artificially annotated description sentence is parsed into a sentence scene graph expressing the objects and their relationships, and the learning of the concept semantic graph is guided and constrained by the sentence scene graph;

[0082] S5) language model decoding:

[0083] The video frame feature sequence is concatenated with the structured coding feature sequence as a visual coding feature sequence of the video to supplement the global and action information of the frame;

[0084] The visual and audio features are respectively input into two Transformer models, and the output features obtained are concatenated to generate a description sentence.

[0085] The method learns the object relationship in the video by using a knowledge graph to realize structured coding to assist video description. First, the set of visual concepts and their relationship weights are screened out by using the knowledge graph to segment the sub-knowledge graph that meets the set. Then, to realize the instantiation of the sub-knowledge graph, i.e., to construct a concept semantic graph for each frame in the video sample, in some instances of the video description method, in step S1:

[0086] The object probability score of each frame in the video frame sequence V = {F1, F2, …, Fn} is detected by a target detection method, and then a probability score matrix S = O x T is obtained to represent the dynamic changes of the objects in the video, wherein S T represents the score of each object in the object set O in the nth frame. n

[0087] The total relationship weight between the same pair of objects is accumulated as the weight of the pair of objects to form an edge matrix A KG ∈R N×N , i.e., if any two nodes <i,j> have a direct edge connected in the graph, then the accumulated sum w ij of the corresponding edge value is obtained, otherwise it is 0, to obtain the concept semantic graph sequence of each video.

[0088] On the basis of the concept semantic graph, the method guides the model to learn the relationship between the objects by the scene semantic information of the video, i.e., the model dynamically learns the relationship that meets the current context from a plurality of relationships of each pair of objects, so as to obtain a concept semantic graph with semantic disambiguation. In some instances of the video description method, in step S2:

[0089] The other objects except the selected object pair <i,j> are used as scene semantics, and the attention mechanism is used to learn the weight of the selected object pair <i,j> in different relationships ,

[0090]

[0091] ​The learned weight value of the object pair <i, j> in the kth relationship Multiplication correspondence relationship The aggregation converts the relationship vector into a new weight value of the object <i, j> wherein, Graph weight; w * : Relationship r * Word vector of the object;

[0092] To realize multi-relation dynamic learning with the updated edge matrix to obtain the concept semantic graph with semantic disambiguation.

[0093] The concept semantic graph sequence is learned through the graph convolution network based on the concept semantic graph to realize the learning reasoning of the concept semantic graph. First, considering that the video has a time sequence characteristic, to learn the frame structured coding feature containing context information, in some instances of video description method, in step S3:

[0094] The score fusion network is used to fuse the video score matrix S, wherein, α n,m represents the influence of the mth frame score S m on the nth frame score S n .

[0095] The fused score matrix is multiplied by the corresponding object word vector as the graph object node feature sequence of the video wherein, represents the fusion score of each object in the nth frame after fusion, represents the graph object node feature matrix of the nth frame represents the word vector of the object set O.

[0096] The graph convolution network model is used to perform structured information propagation on the concept semantic graph to obtain The updated node features of the network are aggregated into the structured coding feature of the frame to finally obtain the structured coding sequence of the video frame

[0097] The functions f(·), g(·), and h(·) are 1x1 convolution networks.

[0098] The relevance is the focus of the evaluation of the description, and the relevance is related to the described thing (video content), i.e. the relevance is related to the objects in the video and their relationships. Therefore, the method proposes a cross-domain guided relationship learning strategy, which parses the annotated description sentence into a sentence scene graph expressing the objects and their relationships, and guides and constrains the learning of the concept semantic graph through the sentence scene graph. The purpose is to avoid the negative effects of the relationship learning caused by the limitations of the cross-entropy loss of the description sentence only, and this process can also to some extent avoid the shortcomings of insufficient learning of the front-end parameters of the model. In some examples of the video description method, in step S4:

[0099] The annotated description sentence of the video i is parsed into a sentence scene graph SG i , and the sentence scene graph SG i is encoded by using a scene graph encoding model to obtain an encoded vector

[0100] The concept semantic graph sequence learned by the graph neural network is fused into the output vector of the video

[0101] The encoded vector is projected into the same space as the output vector of the video i to obtain the transposed sentence scene graph vector and the transposed video concept semantic graph fusion vector , and the similarity of and is taken as part of the loss function to guide and constrain the learning of the concept semantic graph, W1 and W2 are learnable matrices.

[0102] In some examples of the video description method, in step S4, the loss function optimization step includes:

[0103] The description sentence loss adopts a cross-entropy loss, and the calculation formula is as follows:

[0104]

[0105] where w 1:t-1 is the word sequence generated before the time step t; θ is a learnable matrix; w 1:T is the annotated sentence;

[0106] The relationship learning loss has the following calculation formula:

[0107]

[0108] Where dist() is the cosine distance between the description statement scene graph encoding vector and the corresponding video structured encoding feature, i is the i-th video; SG is the description statement scene graph; KG is the video structured encoding;

[0109] Total model loss total It consists of the above two parts:

[0110] loss total =λ1loss sent +λ2loss rel

[0111] Where, λ 1-2 These are hyperparameters representing the proportion of the two losses.

[0112] In some examples of video description methods, in step S5:

[0113] Video frame feature sequence With structured encoded feature sequences Concatenate them to form a visual coding feature sequence for the video. To supplement the global and motion information of the frame;

[0114] Considering that video is multimodal data, audio feature sequences were also added.

[0115] Visual and audio features are input into two Transformer models respectively, and the resulting visual output feature V is then used. d and audio output characteristics A d Concatenate the components to generate a descriptive statement;

[0116] V d =Transformer V (V e )

[0117] A d =Transformer A (A e )

[0118] sent={w1,w2,…,w l} = softmax(FC(concat(V d A d )))

[0119] Among them, Transformer V It is a visual encoder, Transformer. A It is an audio encoder Transformer, concat is feature concatenation, and FC is a fully connected layer.

[0120] In some examples of the video description method, the word vector features are extracted by a TransE model.

[0121] In some examples of the video description method, the audio features are extracted by a VGGish model.

[0122] In some examples of the video description method, the visual features are extracted by an I3D model.

[0123] In some examples of the video description method, the parameters of the graph convolution network model based on the concept semantic graph are set as follows:

[0124] 3 graph convolution layer dimensions are 100, 256, and 512, respectively;

[0125] The audio and visual modes of the Transformer are 128 and 1024, respectively, each using one layer, and H = 4 is used in the multi-head attention block.

[0126] The model uses an Adam optimizer, β = (0.9, 0.99), a smoothing parameter γ = 0.7, and a batch size of 16.

[0127] In order to regularize the weights of the model, the model sets Dropout = 0.1 and acts on the output of the position encoding, before the residual of each sublayer and after the first layer inside the multi-modal generator.

[0128] The above features can be combined in any way without conflict.

[0129] Referring to Figure 1 A video description method based on semantic disambiguation structured coding includes the following steps:

[0130] (1) Feature extraction

[0131] Audio features: The audio features are extracted by a VGGish model. The input of the VGGish model is a 96x64 log mel-scaled spectrogram extracted from a non-overlapping 0.96 second segment. The audio is resampled to 16kHz single-channel audio. The spectrogram is obtained by performing a short-time Fourier transform on the audio using a 25ms Hann time window and a 10ms frame shift. The output is a 128-dimensional feature vector obtained after the activation function of the previous layer before classification.

[0132] Visual features: Visual features are extracted by I3D model. The input of this model is a set of 24 RGB and 25 Fps optical flow frames. Optical flow frames are extracted by PWCNet model. First, adjust the size of each frame so that the short side is 256 pixels, and then crop the center area to get 224x224 frames. The RGB branch and the optical flow branch output and add respectively to generate a 1024-dimensional feature vector.

[0133] Word vector features: Word vector features are extracted by TransE model. TransE model uses OpenKE open source framework, with learning rate of 0.003, optimizer of SGD, and 1000 epochs to obtain word vectors with dimension of 100.

[0134] (2) Conceptual semantic graph construction

[0135] The ConceptNet5.5 knowledge graph is used to screen visual concepts related to the ActivityNet Captions description dataset (select the top 240 entity word groups with the highest frequency from the description dataset corpus to form the object set O = {o1, o2, …, o 240} and their relationship weights to segment the sub-knowledge graph that meets the dataset.

[0136] For graph nodes, the Fasterrcnn target detection method is used to detect the probability score of each object in the video frame sequence V = {F1, F2, …, F 30}, and then obtain the probability score matrix S = O x T to represent the dynamic changes of objects in the video, where S n represents the score of each object in the object set O in the nth frame. For edge matrix A KG ∈R 240×240 , ConceptNet5.5 is a multi-relation graph, i.e. there may be multiple relationships between the same pair of objects, and each relationship has a weight. The method accumulates the total relationship weight between the same pair of objects as the weight of the pair of objects to form the edge matrix, i.e. if any two nodes <i, j> have a direct edge connected in the graph, then is the cumulative sum of the corresponding edge value w ij , otherwise it is 0. Thus, the conceptual semantic graph sequence of each video is obtained.

[0137] (3) Conceptual semantic graph-based graph convolution network

[0138] The conceptual semantic graph sequence is learned by the conceptual semantic graph-based graph convolution network (CS-GCN) to realize the learning and reasoning of the conceptual semantic graph, as shown in the framework. Figure 2 First, the score fusion network is used to fuse the video score matrix S. The fused score matrix Multiplying with the corresponding object word vector as the video's graph object node feature sequence Then, the concept semantic graph is learned by GCN model The structured information propagation is performed to obtain And the updated node features of the network are aggregated into the structured encoding features of the frame. Finally, the video frame structured encoding sequence is obtained

[0139] (4) Concept semantic graph based on semantic disambiguation

[0140] Other objects outside the selected object pair <i,j> As the scene semantics, and the attention mechanism is used to learn the weight of the selected object pair <i,j> in different relationships The learned weight value (Object pair <i,j> weight value in the kth relationship) is multiplied by the corresponding relationship And aggregated and converted into a relationship vector as the new weight value of object <i,j> Finally, the updated edge matrix is used to realize multi-relationship dynamic learning. Figure 3 is a schematic diagram of the effectiveness of the concept semantic graph module. The nodes and edges with lower scores are ignored in the concept semantic graph. The nodes are object names and detection score probabilities, and the edges are relationship names and the maximum weight coefficients learned. The video and its different module description sentences are shown on the top of the diagram, and the concept semantic graph learned based on semantic disambiguation is shown on the bottom.

[0141] (5) Cross-domain guidance relationship learning

[0142] The annotated description sentence of video i is parsed into a sentence scene graph SG by a scene graph language parser i And the SGAE model is used to encode the sentence scene graph SG i The encoded vector is projected into the same space as the CS-GCN output vector of video i And their similarity is calculated as part of the loss function to enhance concept semantic graph learning. Figure 4 is a schematic diagram of the effectiveness of the cross-domain guidance module. The nodes and edges with lower scores are ignored in the concept semantic graph. The video sample, the corresponding generated description sentence, and the learned concept semantic graph are shown on the top of the diagram, and the learning effect with and without the addition of the cross-domain guidance module is shown on the bottom.

[0143] (6) Language model

[0144] The I3D feature sequence is concatenated with the structured encoding feature sequence to obtain the visual encoding feature sequence of the video with global and action information of the supplementary frame. Then, the visual features and audio features are input into two Transformer models respectively, and the obtained output features are concatenated to generate a description sentence. The generated description sentence is compared with the annotated description sentence as shown in Figure 5 . It can be seen from Figure 5 that the video description method of the present application can obtain more accurate and comprehensive description.

[0145] (7) Model parameter setting

[0146] The dimensions of the three graph convolution layers of the CS-GCN module are 100, 256 and 512 respectively. The audio and visual modes of the Transformer are 128 and 1024 respectively, each using one layer, and H=4 is used in the multi-head attention block. The model uses Adam optimizer optimizer, β=(0.9, 0.99), the smoothing parameter γ=0.7, and the batch size is set to 16. In order to regularize the weights of the model, the model sets Dropout=0.1, which acts on the output of the position encoding, before the residual of each sublayer and after the first layer inside the multi-modal generator. During the training process, the learning rate of the model decreases with the increase of the number of iterations, the initial learning rate of the CS-GCN module is 0.001, while the initial learning rate of the Transformer is 0.00001, and every 20 iterations, it is reduced by 0.9 times, and the total number of iterations is 50 epochs.

[0147] The above is a further detailed description of the present application, which cannot be regarded as a specific implementation of the present application. For ordinary skilled persons in the technical field to which the present application belongs, simple deduction or replacement without departing from the concept of the present application is within the protection scope of the present application.

Claims

1. A video description method based on semantic disambiguation structured coding, comprising the following steps: S1) Construct a concept semantic graph: The knowledge graph is used to filter out a set of visual concepts and the weights of the relationships between visual concepts, and then to segment out sub-knowledge graphs that meet the set criteria. Instantiating a sub-knowledge graph involves constructing a concept semantic graph for each frame in a video sample, resulting in a sequence of concept semantic graphs for each video. Specifically, this includes: Select the most frequent words from the descriptive dataset corpus. An object set composed of individual entity words The video frame sequence was obtained by object detection method. The probability score of each object in each frame is used to obtain the probability score matrix. To represent the dynamic changes of object nodes in the video, where, Indicates 1 to The set of objects in the nth frame Scores for each object; Accumulate all relational weights between the same pair of object nodes to form the edge matrix of that pair of object nodes. KG stands for Knowledge Graph, which means that if any two object nodes... If there are directly connected edges in a knowledge graph, then The corresponding boundary values ​​are summed. Otherwise, the value is 0, resulting in a sequence of concept semantic graphs for each video. S2) Construct a concept semantic graph based on semantic disambiguation: Based on the conceptual semantic graph, the scene semantic information of the video guides the model to learn the relationships between semantically disambiguated object nodes. This allows the model to dynamically learn relationships that fit the current context from multiple relationships between each pair of object nodes, thereby obtaining a semantically disambiguated conceptual semantic graph. Specifically, this includes: With selected object node pairs Other object nodes As scene semantics , Indicates the first The feature matrix of object nodes in the frame image is obtained, and the attention mechanism is used to learn the selected object node pairs. In different relationships The weights in "k" represents the k-th relation among K types; the learned object node pairs In the Weight values ​​in a relation Multiplication correspondence Aggregate and convert into relation vectors as object nodes New weight values , ,in, For relationship Weights in a knowledge graph For relationship The word vectors are given new weight values. Update to obtain the edge matrix ; With the updated edge matrix To achieve dynamic learning of multiple relationships and obtain a semantic graph of semantic disambiguation; S3) Learning using graph convolutional networks based on concept semantic graphs: The concept semantic graph sequence is learned through a graph convolutional network based on the concept semantic graph to learn the temporal evolution process, thereby realizing the learning and reasoning of the concept semantic graph and obtaining the structured encoded sequence of video frames. Specifically, it includes: Using a score fusion network to fuse video score matrices , ,in, Indicates the first Frame score For the Frame score The impact, among which and Indicates different frames from 1 to T; The merged score matrix Multiplying the corresponding word vectors by the graph object vectors yields the feature sequence of the video's nodes. , ,in, Indicates the first Feature matrix of object nodes in frame-to-frame image , Representation of object set Word vectors, Indicates the fusion of the first The fusion score of each object in the frame; Through graph convolutional network model Methods for concept semantic graphs To obtain structured information dissemination , The updated node features of the network are aggregated into the structured coded features of the frame. Finally, the structured encoded sequence of video frames is obtained. ; function , , It is a 1 × 1 convolutional network; S4) Cross-domain mentoring relationship learning: Manually annotated descriptive statements are parsed into statement scenario graphs that express objects and their relationships. These statement scenario graphs are then used to guide and constrain the learning of concept semantic graphs. Specifically, this includes: Video The annotation description statement is parsed into a statement scenario diagram. SG represents the scenario graph describing the statements, and the scenario graph is used to encode the model. Method to Statement Scenario Diagram Encode the vector to obtain the encoded output vector. , ; The sequence of concept semantic graphs learned by the graph neural network is fused into the output vector of the video. ; Encoded vector With video The output vector Projecting them into the same space yields the transposed statement scene graph vector. Fusion vector with transposed video concept semantic graph And seek and The similarity is used as part of the loss function to guide and constrain the learning of the concept semantic graph. ; , ; and It is a learnable matrix; S5) Language model decoding: Visual and audio features are input into two Transformer models respectively, and the resulting output features are concatenated to generate a descriptive statement, specifically including: Video frame feature sequence With structured encoded feature sequences Concatenate them to form a visual coding feature sequence for the video. This is to supplement the global and motion information of the frame; Considering that video is multimodal data, audio feature sequences were also added. ; Visual and audio features are input into two Transformer models respectively, and the resulting visual output features are then processed. and audio output characteristics Concatenate the components to generate a descriptive statement; ， ， ， in, It is a visual encoder, Transformer. It is an audio encoder, Transformer. It is feature concatenation. It is a fully connected layer; In steps S4 and S5, the loss function optimization steps include: The loss statement used is the cross-entropy loss, calculated as follows: ， in, It is a time step The previously generated word sequence, Representing word vectors, It is a learnable matrix; The formula for calculating relation learning loss is as follows: ， in, It is the cosine distance between the scene graph encoding vector of the description statement and the corresponding video structured encoding features; Total model loss It consists of the above two parts: ， in, and These are hyperparameters representing the proportion of the two losses.

2. The video description method according to claim 1, characterized in that, Word vector features are extracted using the TransE model.

3. The video description method according to claim 1, characterized in that, Extracting audio features using the VGGish model; and / or Visual features are extracted using the I3D model.

4. The video description method according to claim 1, characterized in that, The parameters of the graph convolutional network model based on the concept semantic graph are set as follows: The dimensions of the three graph convolutional layers are 100, 256, and 512, respectively. The Transformer has 128 audio and 1024 visual modes, each using one layer and a multi-head attention block. ; The model uses the Adam optimizer. Smoothing parameters Batch size set to 16; To regularize the model's weights, the model is set... It is applied to the output of the position encoding, before the residual of each sub-layer and after the first layer inside the multimodal generator.

Images