Video common sense text generation method based on unstructured knowledge embedding

Abstract

The application discloses a video common-sense text generation method based on unstructured knowledge embedding, comprising the following steps: S1, extracting image features, video features and audio features of a video, and generating a text description of the video content through cross-modal fusion and decoding processing; S2, inputting the text description of the video content into a pre-trained unstructured knowledge base GPT, and obtaining a deep common-sense description through a pre-set prompt word; S3, performing screening processing on the expanded text knowledge, and effectively fusing the image features, the video features and the audio features of the video with the cross-modal common sense, so as to finally generate a common-sense description text which can accurately understand the connotation of the video. The application proposes a learning method of multi-modal knowledge introduction, and realizes the deep understanding of the video content of the model from the two aspects of knowledge expansion and knowledge fusion by means of a large-scale unstructured pre-training model, which is helpful to the video-based text description generation task in the Internet era.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vision-based cross-modal text generation, and particularly relates to a video common-sense text generation method based on unstructured knowledge embedding. BACKGROUND

[0002] There are a large number of videos in massive Internet data. In order to quickly obtain the knowledge in these videos, video-based text generation tasks have been widely concerned in the academic and industrial circles. When people watch a video, they can not only understand the events occurring in the video, but also understand the deep meanings behind the events in the video. For example, a video tells the story of "a person is making breakfast". When watching the video, a human being can further reason that the purpose of the person in the video is "he wants to eat healthy food", the impact of the event in the video is "he can have a big meal soon", and the person making breakfast is "a person with a good hand". Therefore, it is urgent to study an effective text generation model which can perform deep reasoning on the content of the video and generate deep common-sense descriptions of the video from the perspectives of the purpose, impact and character attributes of the event in the video.

[0003] For this task, existing methods often try to generate common-sense descriptions by constructing the association between vision and unstructured entities. They usually adopt a model architecture based on a coder-decoder. First, the video is encoded into a feature vector by an encoder, and then the video features are decoded by a decoder to generate descriptions of the content of the video and common-sense descriptions containing deep semantics. However, these methods ignore a problem: the common-sense descriptions to be generated by the target often do not appear in the form of visual entities in the video. In addition, videos contain rich modal information such as images and audio, and existing methods are difficult to effectively realize the feature fusion among these multi-modal information. SUMMARY

[0004] The purpose of the present application is to provide a video common-sense text generation method based on unstructured knowledge embedding, which selectively searches and filters knowledge information related to the video in an unstructured knowledge base, generates common-sense descriptions of the video in a knowledge expansion and knowledge fusion manner, and thus realizes deep understanding of the video.

[0005] Technical scheme: To solve the above technical problems, the technical scheme adopted by the present application is:

[0006] In a first aspect, a video common-sense text generation method based on unstructured knowledge embedding is provided, comprising:

[0007] Step S1: extracting image features, video features and audio features of the video, and generating a text description of the video content through cross-modal fusion and decoding processing;

[0008] Step S11, extracting image features, video features and audio features of the video;

[0009] Step S12, cross-modal fusion of the image features, video features and audio features of the video to obtain multi-modal features;

[0010] Step S13, decoding the multi-modal features through a pre-trained video content decoder to generate a text description of the video content;

[0011] Step S2: inputting the text description of the video content into a pre-trained unstructured knowledge base GPT to obtain a deep-level common sense description through a pre-set prompt word;

[0012] Step S3: after encoding feature extraction of the deep-level common sense description, cross-modal common sense fusion of the image features, video features and audio features of the video to obtain multi-modal common sense features, and decoding the multi-modal common sense features to generate a common sense description text containing deep-level connotations of the video;

[0013] Step S31, encoding feature extraction of the deep-level common sense description by using a Bert model to obtain text common sense features;

[0014] Step S32, cross-modal common sense fusion of the text common sense features and the image features, video features and audio features of the video to obtain multi-modal common sense features;

[0015] Step S33, decoding the multi-modal common sense features by using a video deep-level common sense decoder to generate a common sense description text containing deep-level connotations of the video.

[0016] In some embodiments, step S11, extracting image features, video features and audio features of the video, comprises:

[0017] extracting the image features of the video by using a pre-trained ResNet152 network structure coding;

[0018] extracting the video features of the video by using a pre-trained I3D network structure coding;

[0019] extracting the audio features of the video by using a pre-trained SoundNet network structure coding.

[0020] In some embodiments, step S12, cross-modal fusion of the image features, video features and audio features of the video to obtain multi-modal features, comprises:

[0021] after encoding the image features, video features and audio features by multiple LSTMs respectively, splicing to obtain a spliced multi-modal feature vector F′ video :

[0022] F′ video =LSTM(FC(V) 3D ))+LSTM(FC(I 2D ))+LSTM(FC(A 1D ))

[0023] Among them, F′ video V represents the concatenated multimodal feature vector; 3D It is a video feature; I 2D It is an image feature; A 1D These are audio features; FC stands for Fully Connected Network Layer, and LSTM stands for Long Short-Term Memory Neural Network.

[0024] The multimodal feature vector F′, which is concatenated, is generated using an encoder based on the Transformer architecture. video The encoding is re-fused to obtain the final multimodal features F. video The encoder based on the Transformer structure includes a set of self-attention layers and feedforward network layers.

[0025] The self-attention layer is represented as follows:

[0026]

[0027] The concatenated multimodal feature vector F′ video Through three mapping matrices W respectively Q W K and W V This yields three input matrices Q, K, and V; d k is the normalization parameter; softmax is the normalized activation function, and T represents the transpose matrix.

[0028] In some embodiments, in step S13, the video content decoder adopts a Transformer-based decoder, which includes, in sequence: a Self-attention layer, a Cross-attention layer, and a Feedforward network layer.

[0029] The Self-attention layer is used to normalize the text modal features;

[0030] The cross-attention layer is used to process the cross-modal association between text and video encoding. By characterizing the cross-modal association between text and video features, it generates the word at the current position that best represents the video content.

[0031] The feedforward network layer is arranged after the Cross-attention layer and is composed of a double-layer fully connected layer, and the expression of each word is strengthened through an activation function, the standardization of each word feature is processed, and the data is mapped to a high-dimensional space and then mapped to a low-dimensional space to learn more abstract text expression;

[0032] The video content decoder training loss adopts a cross-entropy loss function, and when each word is generated, the word with the maximum appearance probability in the next position is predicted according to the previously generated words;

[0033] The video content decoder loss function is

[0034]

[0035] Wherein, N cap is the total length of the generated text description, y t represents the word that should be generated at the t position, F video is a multi-modal feature vector, θ cap is a model parameter, and p() represents a mapping probability;

[0036] The generated words are spliced to obtain a sentence S cap of the video content text description, and the formula is as follows:

[0037]

[0038] In some embodiments, in step S2, the training method of the unstructured knowledge base GPT includes:

[0039] The video content text description and the deep-level common sense description are connected through a preset prompt word to form a long sample for introducing the unstructured knowledge base GPT training; wherein the long sample is composed of three parts: <video content text description, prompt word, deep-level common sense description>;

[0040] When training the unstructured knowledge base GPT, the video content text description and the prompt word are used as known information, and the unstructured knowledge base GPT model generates the deep-level common sense description word by word, and the training loss function L gpt is as follows:

[0041]

[0042] Wherein, N is the total sample length; token n is a word that needs to be generated by GPT during training, and token n-Krepresents the existing words input to the GPT; K represents the total length of the sentence of the text description of the video content and the prompt word; θ gpt is the model parameter; P() represents the mapping probability.

[0043] In some embodiments, in step S2, the prompt word includes:

[0044] For the purpose of the event in the video, set the prompt word "the aim is to:";

[0045] For the impact of the event in the video, set the prompt word "the effect is:";

[0046] For the attribute of the person or event in the video, set the prompt word "the person or event is:".

[0047] In some embodiments, in step S32, the text common sense feature is cross-modal common sense fused with the image feature, the video feature and the audio feature of the video to obtain a multi-modal common sense feature, including:

[0048] The feature obtained by Bert encoding is mapped to a semantic space that can be shared with the video feature through feature mapping, and then it is cross-modal common sense fused with the three-dimensional features in the video, and the formula is as follows:

[0049]

[0050] wherein, is the fused multi-modal common sense feature, FC is a fully connected network layer, LSTM is a long short-term memory neural network, V 3D is a video feature; I 2D is an image feature; A 1D is an audio feature; S cms represents a common sense description text, and Bert represents an encoding by a Bert model.

[0051] In some embodiments, in step S33, the video deep common sense decoder adopts a Transformer-based decoder, which includes, in sequence, a Self-attention self-attention layer, a Cross-attention cross-attention layer and a Feedforward feedforward network layer.

[0052] The Self-attention layer is used for normalizing the text modal feature.

[0053] Cross-attention layer, used to process cross-modal information, compares the association between the word and the video and the external common sense introduced by GPT at each generated word;

[0054] Feedforward layer, which is set after the Cross-attention layer, maps through the Feedforward layer to find in the word library in the form of probability to generate the final common sense description text;

[0055] The video deep common sense decoder uses a cross-entropy function as the training loss, generates the word with the highest probability at the current position at each iteration step, and generates a common sense description text word by word; As follows:

[0056]

[0057] Where N cms is the total length of the common sense description text, y t represents the word that should be generated at position t, is the multimodal common sense feature, S cap is the text description of the video content, θ cms is the decoder model parameter, and p() represents the mapping probability.

[0058] In some embodiments, the total loss is composed of two parts: the video content decoder loss function and the video deep common sense decoder loss function:

[0059]

[0060] Wherein, is the video content decoder loss function, is the video deep common sense decoder loss function.

[0061] In a second aspect, the present application provides a video common sense text generation device based on unstructured knowledge embedding, comprising a processor and a storage medium;

[0062] The storage medium is used to store instructions;

[0063] The processor is used to operate according to the instructions to perform the steps of the method according to the first aspect.

[0064] In a third aspect, the present application provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method according to the first aspect.

[0065] The application has the advantages that the application can not only automatically generate the description of the event content of the video itself, but also generate the common-sense description of the video from three angles respectively, so as to realize the deep understanding of the video, for the known video. The common-sense description specifically includes the intention of the character or event in the video, the effect of the event, and the internal attribute and characteristics of the character. For example, the picture described in a video is "a person is giving a speech on the stage", and the application aims to make the computer generate descriptions such as "the audience on the stage will benefit a lot", "the speaker is knowledgeable", so that the computer can truly recognize the connotation behind the video. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 The method flowchart of the embodiment of the application is shown.

[0067] Figure 2 The text description generation block diagram of the video content in the embodiment of the application is shown.

[0068] Figure 3 The knowledge expansion block diagram based on the unstructured knowledge base in the embodiment of the application is shown.

[0069] Figure 4 The cross-modal common-sense feature fusion method block diagram in the embodiment of the application is shown.

[0070] Figure 5 The video deep-level common-sense description text generation schematic diagram in the embodiment of the application is shown. DETAILED DESCRIPTION

[0071] In order to make the technical means, creative features, purposes and effects realized by the application easy to understand, the application is further described below in combination with specific embodiments.

[0072] In the description of the application, if the meaning of several is more than one, the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, above, below, within, etc. are understood as including the number. If it is described as first, second, it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0073] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "illustrative embodiment", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0074] The technical problems to be solved by the present application are mainly: deep reasoning of deep-level knowledge that does not exist in the video; how to effectively introduce associated common sense from an unstructured knowledge base;

[0075] Further, it can also include how to effectively remove irrelevant redundant knowledge and realize multi-modal feature fusion for rich multi-modal information such as images and audios in the video.

[0076] In order to solve the above technical problems, the present application proposes a method for generating video common sense text based on unstructured knowledge embedding, which selectively searches and filters knowledge information related to the video in the unstructured knowledge base, generates the common sense description of the video in the way of knowledge expansion and knowledge fusion, so as to realize the deep understanding of the video.

[0077] As described above, the generation task of the common sense description based on the video needs the model to perform deep reasoning on the events in the video. While generating the description of the content of the video event, the common sense description of the video is generated from three angles of the intention of the characters or events in the video, the effect of the event and the internal attribute and characteristics of the characters. Since the content of the three types of common sense description often does not appear in the form of visual entity in the video, it is impossible to effectively generate the accurate common sense description of the video only by relying on the visual information in the video. Therefore, the present application proposes a method and system that can embed unstructured knowledge, as shown in Figure 2 The method obtains the knowledge of text modal from the unstructured knowledge base, deepens the understanding of the model to the video. In addition, the introduced unstructured knowledge may also have problems such as information redundancy. Therefore, the present application also needs to filter the expanded knowledge and effectively perform cross-modal fusion with the existing video features, and finally generate the common sense description that can accurately understand the connotation of the video.

[0078] Since there is a cross-modal semantic gap problem between video and text, the application first needs to encode the video containing complex modal information to map into a multi-modal shared semantic space; then, since deep knowledge often does not exist in the feature space of the video, the application searches in the unstructured knowledge base to obtain common sense knowledge. By setting three fixed prompts for three different common sense and modifying the corresponding data training format to train GPT respectively. By directional retrieval in the unstructured resource library, the common sense information that does not exist in the video is obtained; finally, since GPT is a text modal based resource library, it cannot well integrate multi-modal information such as image, video, audio and the like. In addition, not all knowledge retrieved from the unstructured resource library is valid knowledge, and there may be certain redundant information. Therefore, the knowledge obtained by GPT is encoded again by Bert to further extract the text knowledge, and the text knowledge is cross-modal fused with the multi-modal features in the video to generate the final common sense description.

[0079] Therefore,

[0080] Embodiment 1

[0081] A video common sense text generation method based on unstructured knowledge embedding, comprising:

[0082] Step S1: extracting image features, video features and audio features of the video, and generating text description of the video content through cross-modal fusion and decoding processing;

[0083] Step S11, extracting image features, video features and audio features of the video;

[0084] Step S12, cross-modal fusion of the image features, video features and audio features of the video to obtain multi-modal features;

[0085] Step S13, decoding the multi-modal features through the pre-trained video content decoder to generate the text description of the video content;

[0086] Step S2: inputting the text description of the video content into the pre-trained unstructured knowledge base GPT to obtain deep common sense description through the pre-set prompt word guide;

[0087] Step S3: cross-modal common sense fusion of the encoded feature extracted from the deep common sense description and the image features, video features and audio features of the video to obtain multi-modal common sense features, and decoding the multi-modal common sense features to generate the common sense description text containing deep connotation of the video;

[0088] Step S31, encoding feature extraction of the deep common sense description by using the Bert model to obtain text common sense features;

[0089] Step S32, cross-modal common sense fusion of the text common sense features and the image features, the video features and the audio features of the video to obtain multi-modal common sense features;

[0090] Step S33, decoding the multi-modal common sense features by using a video deep-level common sense decoder to generate a common sense description text containing deep-level connotations of the video.

[0091] In some embodiments, step S11, extracting the image features, the video features and the audio features of the video, comprises:

[0092] extracting the image features of the video by using a pre-trained ResNet152 network structure coding;

[0093] extracting the video features of the video by using a pre-trained I3D network structure coding;

[0094] extracting the audio features of the video by using a pre-trained SoundNet network structure coding.

[0095] In some embodiments, step S12, cross-modal fusion of the image features, the video features and the audio features of the video to obtain multi-modal features, comprises:

[0096] concatenating the image features, the video features and the audio features after coding by a plurality of LSTMs to obtain a concatenated multi-modal feature vector F′ video :

[0097] F′ video = LSTM(FC(V 3D ))+LSTM(FC(I 2D ))+LSTM(FC(A 1D )

[0098] wherein F′ video represents the concatenated multi-modal feature vector; V 3D is the video features; I 2D is the image features; A 1D is the audio features; FC is a fully connected network layer, and LSTM is a long short-term memory neural network;

[0099] re-fusing and coding the concatenated multi-modal feature vector F′ video by using an encoder based on a Transformer structure to obtain final multi-modal features F video ; wherein the encoder based on the Transformer structure comprises a group of Self-attention self-attention layers and Feedforward feedforward network layers;

[0100] The self-attention layer is represented as follows:

[0101]

[0102] The spliced multi-modal feature vector F' is obtained video through three mapping matrices W Q , W K and W V , respectively, to obtain three input matrices Q, K and V; d k is a normalization parameter; softmax is a normalization activation function, and T represents a transposed matrix.

[0103] In some embodiments, in step S13, the video content decoder employs a Transformer-based decoder, which sequentially includes a Self-attention layer, a Cross-attention layer and a Feedforward network layer.

[0104] The Self-attention layer is used for normalizing the text modal features.

[0105] The Cross-attention layer is used for processing the cross-modal association between the text and the video encoding, and generates a word that best represents the video content at the current position by depicting the cross-modal association between the text and the video features.

[0106] The Feedforward network layer is arranged after the Cross-attention layer and is composed of a double-layer fully connected layer, which strengthens the expression of each word through an activation function, performs standardization processing on each word feature, and maps the data from a high-dimensional space to a low-dimensional space to learn more abstract text expressions.

[0107] The video content decoder training loss employs a cross-entropy loss function, which predicts the word with the maximum appearance probability at the next position according to the previously generated words when generating each word.

[0108] The video content decoder loss function is represented as follows:

[0109]

[0110] where N cap is the total length of the target generated text description, y t represents the word that should be generated at the t position, F video is the multi-modal feature vector, and θ capp() represents the mapping probability for model parameters;

[0111] The generated words are spliced to obtain a sentence S of the text description of the video content cap , as follows:

[0112]

[0113] In some embodiments, in step S2, the training method of the unstructured knowledge base GPT includes:

[0114] The text description of the video content and the deep-level common sense description are connected through a preset prompt word to form a long sample for introducing the unstructured knowledge base GPT training; wherein the long sample is composed of three parts: <text description of video content, prompt word, deep-level common sense description>;

[0115] When training the unstructured knowledge base GPT, the text description of the video content and the prompt word are used as known information, and the unstructured knowledge base GPT model generates a deep-level common sense description word by word, and the training loss function L gpt is as follows:

[0116]

[0117] Wherein, N is the total sample length; token n is the word that GPT needs to generate during training, token n-K represents the existing word input to GPT; K represents the total length of the sentence of the text description of the video content and the prompt word; θ gpt is the model parameter; P() represents the mapping probability.

[0118] In some embodiments, in step S2, the prompt word includes:

[0119] For the purpose of the event in the video, set the prompt word "the aim is to:";

[0120] For the impact of the event in the video, set the prompt word "the effect is:";

[0121] For the attribute of the person or event in the video, set the prompt word "the person or event is:".

[0122] In some embodiments, in step S32, the text common sense feature is fused with the image feature, video feature and audio feature of the video to obtain a multi-modal common sense feature, including:

[0123] The Bert encoded features are mapped to a semantic space that can be shared with the video features through feature mapping, and then fused with the three-dimensional features in the video through cross-modal common sense fusion, which can be expressed as follows:

[0124]

[0125] wherein, is the fused multi-modal common sense feature, FC is a fully connected network layer, LSTM is a long short-term memory neural network, V 3D is the video feature; I 2D is the image feature; A 1D is the audio feature; S cms represents the common sense description text, and Bert represents the encoding through the Bert model.

[0126] In some embodiments, in step S33, the video deep common sense decoder adopts a Transformer-based decoder, which sequentially includes a Self-attention self-attention layer, a Cross-attention cross-attention layer, and a Feedforward feedforward network layer.

[0127] The Self-attention layer is used for normalizing the text modal feature.

[0128] The Cross-attention layer is used for processing cross-modal information, and compares the association between the word and the video and the external common sense introduced by GPT at each generated word.

[0129] The Feedforward feedforward network layer is arranged after the Cross-attention layer, and maps through the Feedforward layer to find in the word library in the form of probability to generate the final common sense description text.

[0130] The video deep common sense decoder adopts a cross-entropy function as the training loss, generates the word with the maximum probability at the current position at each iteration step, and generates a common sense description text word by word. As follows:

[0131]

[0132] wherein, N cms is the total length of the common sense description text, y t represents the word that should be generated at position t, is the multi-modal common sense feature, S cap is the text description of the video content, and θ cmsHere are the decoder model parameters, and p() represents the mapping probability.

[0133] In some embodiments, the total training loss It consists of two parts: the video content decoder loss function and the video deep commonsense decoder loss function.

[0134]

[0135] in, The loss function for the video content decoder. This is the loss function for the deep commonsense decoder of video.

[0136] In some embodiments, such as Figure 1 As shown, the method in this embodiment includes three steps: S1, extracting multimodal information contained in the video to generate a text description of the video content; S2, expanding the text description of the video content by acquiring knowledge of the text modality from an unstructured knowledge base; S3, filtering and processing the expanded knowledge, and generating a common sense description containing the deep connotation of the video by performing cross-modal fusion of the multimodal information in the video.

[0137] The following is a detailed introduction:

[0138] S1. Extract multimodal information from the video to generate a text description of the video content.

[0139] The method for extracting multimodal information from videos and generating video content descriptions proposed in this invention is as follows: Figure 2 The process consists of three steps, detailed below:

[0140] S11: Extract multimodal features from the video

[0141] For a given video, this invention first encodes it into a semantically rich shared space. Specifically, this invention uses a set of pre-trained ResNet152, I3D, and SoundNet network structures to encode the image features, video features, and audio features in the video into feature vectors in the shared semantic space, respectively.

[0142] S12: Fusion of multimodal semantic features

[0143] Due to the cross-modal semantic gap problem between features of different modalities, the application proposes a cross-modal fusion method. By constructing a semantic space that can be shared within multiple modalities, the application fuses multi-modal features. Specifically, we first encode the features extracted by pre-training within the modalities, and map them to a fixed-dimensional modal space through a linear function. Then the application obtains the feature representation within the modalities by multiple LSTMs respectively encoding the features of different modalities. The specific formula is as follows:

[0144] F′ video =LSTM(FC(V 3D ))+LSTM(FC(I 2D ))+LSTM(FC(A 1D ))

[0145] Wherein, F′ video represents the multi-modal feature vector after splicing; V 3D is the three-dimensional video feature extracted by the I3D network; I 2D is the two-dimensional image feature extracted by ResNet152; A 1D is the one-dimensional audio feature extracted by SoundNet. FC is a fully connected network layer.

[0146] In addition, the application introduces an encoder based on the Transformer structure. Through a set of Self-attention layers and Feedforward layers, the application re-fuses and encodes the multi-modal features to obtain a feature code that can represent across modalities. The specific formula of the Self-attention layer is as follows:

[0147]

[0148] The application maps the multi-modal feature vector F′ video through three mapping matrices W Q , W K and W V to obtain Q, K and V in the above formula; d k is a normalization parameter; softmax is a normalization activation function. After the operation of multiple layers and multiple attention, the application finally generates the interactive multi-modal feature F video that can represent each other in the shared semantic space.

[0149] S13: generating text description based on video content

[0150] For the multi-modal features extracted in S12, the application generates a text description of the video event itself through a decoder. The application uses a decoder based on a Transformer, and the training loss uses a cross-entropy function. When generating each word, the word with the highest probability of appearing at the next position is predicted according to the previously generated words. The loss function is as follows:

[0151]

[0152] Where N cap is the total length of the generated text description, y t represents the word that should be generated at position t, F video is the video feature vector spliced in S12, and θ cap is the model parameter.

[0153] Finally, the generated words are spliced to obtain the sentence S cap of the text description of the video content, and the formula is as follows:

[0154]

[0155] The video content decoder of the present application is composed of three parts, namely, a Self-attention layer, a Cross-attention layer and a Feedforward network layer. The Self-attention layer processes information of the pure text mode. When generating each word, the Self-attention layer compares the correlation of the word with the previously generated words to determine what type of word is more suitable for the current position from the perspective of text fluency. In the present application, the model structure of the Self-attention layer is the same as the formula of the Self-attention layer in step S12, wherein the three matrices Q, K and V are all features of the text mode. Subsequently, the Cross-attention layer is used to process the cross-modal association between the text and the video encoding. The model function of the Cross-attention layer is the same as that of the Self-attention layer, except that in the Cross-attention layer, only the Q matrix is the feature of the text mode, and the K and V matrices are both features of the video mode. The Cross-attention generates a word that best represents the video content at the current position by depicting the cross-modal association between the text and the video features. Finally, the present application introduces a Feedforward layer after the Cross-attention layer, which is composed of a double-layer fully connected layer. Although the features of each word can be obtained through the Self-attention and Cross-attention layers, the expression ability is not strong. We strengthen the expression of each word through the activation function. Through the standardization processing of each word feature, the data is first mapped to a high-dimensional space and then mapped to a low-dimensional space to learn more abstract text expression. The decoder of the present application generates an accurate text description corresponding to the video under the help of the above three-layer network structure.

[0156] S2: obtaining knowledge of the text mode from the unstructured knowledge base, and expanding the text description of the video content

[0157] For the text description generated by S1, it is found that the generated description cannot effectively interpret the deep-level knowledge contained in the video.

[0158] The present application designs a method of introducing unstructured knowledge to solve the problem, which selectively introduces knowledge associated with the video content from an external knowledge base to complete the deep understanding of the video, as shown in Figure 3 The specific steps are as follows:

[0159] S21, setting of prompt

[0160] Although the external unstructured knowledge base (GPT) contains a large amount of knowledge, most of the knowledge is irrelevant to the video. Therefore, the present application designs different prompts for different knowledge, and retrieves the knowledge base to obtain more relevant common sense descriptions of the video content.

[0161] Regarding the unstructured knowledge base, the present application uses the GPT model to obtain the knowledge contained therein by fine-tuning the GPT. The goal of the present application is to obtain deep knowledge contained in the video. These knowledge may not be in the form of visual entities in the video, but they are still associated with the video to some extent. In addition, deep knowledge often exists in various forms, such as the purpose of an event in the video, the impact, and the attributes of the event. Different prompts are set for different deep knowledge. For example, for the purpose of an event in the video, the present application sets the prompt "the aim is to:"; for the impact of the event in the video, the present application sets the prompt "the effect is:"; for the attributes of the person or event in the video, the present application sets the prompt "the person or event is:".

[0162] S23, introducing an unstructured knowledge base (GPT) to realize knowledge expansion

[0163] Although the prompts are set in S22, in order to facilitate the training of GPT, the present application further modifies the data format in the training set. Specifically, the present application connects the text description of the video content and the deep common sense description through the prompt to form a long sample for GPT training. Therefore, the long sample is composed of three parts: <text description of video content, prompt, deep common sense description>. When training GPT, the text description of the video content and the prompt are used as known information, and the model generates the deep common sense description word by word, and the training loss is as follows:

[0164]

[0165] Wherein, N is the total sample length; token n is the word that GPT needs to generate during training, token n-K represents the existing word input to GPT; K represents the total length of the text description of the video content and the prompt; θ gpt is the model parameter.

[0166] Furthermore, even for the same type of common sense, there are many descriptions that accurately match the video. For example, in a video, "a person is singing," the purpose of the event could be "cultivating one's character" or "preparing for a performance." Although these common sense contents are different, they all correspond to the video "a person is singing." Therefore, in order to enable GPT to learn multiple types of knowledge simultaneously, this invention incorporates multiple common sense sentences of the same type but with different semantics when constructing the deep common sense description part of the training set samples; and separates them with the "#" marker during training. This allows this invention to obtain richer knowledge when searching for information in GPT.

[0167] S3: The expanded text knowledge is filtered and processed, and then fused with video features across modalities to generate a common-sense description containing the deep meaning of the video.

[0168] Through the knowledge expansion step in S2, we can obtain common-sense descriptions related to the video content from the unstructured knowledge base. However, GPT is a resource base primarily based on text modality, and it cannot effectively integrate other modalities such as images, videos, and audio. Furthermore, not all knowledge retrieved from external resource bases is valid common-sense; some redundant information may exist.

[0169] This invention addresses this problem by designing a cross-modal knowledge fusion method that generates accurate common-sense descriptions with the help of expanded knowledge. Specifically, it consists of the following three steps:

[0170] S31, Feature Encoding of Text Common Sense Modality

[0171] To address the knowledge generated by GPT in S2, this invention first encodes it using the BERT model to extract its rich semantic knowledge. Furthermore, to prevent excessive interference from the common sense generated in S2 during model training, this invention selects only the features at the CLS position as the encoded features after encoding with BERT.

[0172] S32, Cross-modal commonsense fusion

[0173] To address the semantic gap between different modalities, this invention, based on the concept of cross-modal fusion, integrates knowledge from the text modality with information from other modalities such as images and audio in videos. Figure 4 As shown. Specifically, this invention obtains the overall features of the video by fusing the textual common sense extracted in S31 with the image features, video features, and audio features in S1. This invention first maps the features obtained by BERT encoding to a semantic space that can be shared with the video features through feature mapping, and then fuses it with the three-dimensional features in the video using a multimodal method, as shown in the following formula:

[0174]

[0175] in, For the fused multimodal common sense features, V 3D I 2D and A 1D S represents the video, image, and audio features extracted from the video, respectively. cms This refers to the common sense description text obtained from GPT in step S22.

[0176] S33, Generation of Common Sense Text

[0177] This invention, based on the Transformer decoder framework, further generates common-sense descriptive text that allows for in-depth interpretation of videos, such as... Figure 5 As shown. Similar to step S13, we still use the cross-entropy function as the training loss when generating common sense text. In each iteration step, we generate the word with the highest probability at the current position, and generate a common sense description text for each word. The loss function is as follows:

[0178]

[0179] Where, N cms The total length of common sense text. For multimodal common sense features, S cap θ is the text description of the video content generated in step S13. cms These are the decoder model parameters.

[0180] The video deep commonsense decoder (second decoder) model consists of three parts: a Self-attention layer, a Cross-attention layer, and a Feedforward layer. The Self-attention layer processes textual modal information, comparing the relevance of each generated word to previous words. The Cross-attention layer processes cross-modal information, comparing the association between each generated word and external commonsense introduced by the video and GPT. Finally, the Feedforward layer maps the words in a probabilistic manner to the vocabulary corpus to generate the final commonsense description text.

[0181] Total training loss of the present invention It consists of two parts: the video content decoder loss function and the video deep commonsense decoder loss function.

[0182]

[0183] in, Let the loss function be the video content decoder. Video deep common sense decoder loss function.

[0184] Application examples:

[0185] The application adopts examples to prove that the generated video common sense description is effective. We aim at the Videl2Commonsense data set, which contains 1000 videos. Among them, we extract 700 videos as the training set, and the other 300 as the test set. We generate the corresponding text description of the video content and various deep common sense content description for each video. The video deep common sense text description includes: the purpose of the event in the video, the impact of the event in the video, and the attributes of the event or the characters in the video. Experiments prove that the multi-modal knowledge representation extracted by the application is accurate and meaningful.

[0186] Embodiment 2

[0187] In a second aspect, the embodiment provides a video common sense text generation device based on unstructured knowledge embedding, comprising a processor and a storage medium;

[0188] The storage medium is used to store instructions;

[0189] The processor is used to operate according to the instructions to perform the steps of the method according to embodiment 1.

[0190] Embodiment 3

[0191] In a third aspect, the embodiment provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method of embodiment 1.

[0192] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0193] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block or blocks. Figure 1 one or more flow or flows and / or block or blocks.

[0194] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block or blocks. Figure 1 one or more flow or flows and / or block or blocks.

[0195] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block or blocks. Figure 1 one or more flow or flows and / or block or blocks.

[0196] Within the scope of the application, all changes or modifications that can be made to the above-described embodiments should be covered by the present application.

Claims

1. A method for generating video commonsense text based on unstructured knowledge embedding, characterized in that, include: Step S1: Extract image features, video features, and audio features from the video, and generate a text description of the video content through cross-modal fusion and decoding. Step S11: Extract image features, video features, and audio features from the video; Step S12: Cross-modal fusion of image features, video features, and audio features of the video to obtain multimodal features, including: encoding image features, video features, and audio features separately using multiple LSTMs and then concatenating them to obtain a concatenated multimodal feature vector; re-fusing and encoding the concatenated multimodal feature vector using an encoder based on a Transformer structure to obtain the final multimodal features; wherein the encoder based on a Transformer structure includes a set of self-attention layers and feedforward network layers; Step S13: Decode the multimodal features using a pre-trained video content decoder to generate a text description of the video content. The video content decoder is a Transformer-based decoder, comprising: a Self-attention layer, a Cross-attention layer, and a Feedforward network layer. The Self-attention layer normalizes the text modal features. The Cross-attention layer handles the cross-modal association between text and video encoding, generating the word that best represents the video content at the current position by characterizing the cross-modal association between text and video features. The Feedforward network layer, located after the Cross-attention layer, consists of a two-layer fully connected layer. It strengthens the expression of each word through an activation function, standardizes the features of each word, and maps the data first to a high-dimensional space and then to a low-dimensional space to learn a more abstract text expression. When generating each word, predict the word with the highest probability of appearing at the next position based on the previously generated words. Concatenate the generated words to obtain the sentence describing the text content of the video content. Step S2: Input the text description of the video content into the pre-trained unstructured knowledge base GPT. Guided by pre-set prompts, obtain a deeper common sense description. The training method of the unstructured knowledge base GPT includes: connecting the text description of the video content and the deeper common sense description through pre-set prompts to form a long sample for training the unstructured knowledge base GPT. The long sample consists of three parts: <text description of video content, prompts, and deeper common sense description>. When training the unstructured knowledge base GPT, the text description of the video content and the prompts are used as known information, allowing the unstructured knowledge base GPT model to generate deeper common sense descriptions word by word. The prompts include: "the aim is to:" for the purpose of the event in the video; "the effect is:" for the impact of the event in the video; and "the person or event is:" for the attributes of the person or event in the video. Step S3: After extracting the encoded features of the deep common sense description, perform cross-modal common sense fusion with the image features, video features and audio features of the video to obtain multimodal common sense features. Decode the multimodal common sense features to generate common sense description text containing the deep connotation of the video. Step S31: Use the BERT model to extract encoding features from the deep common sense description to obtain text common sense features; Step S32: Perform cross-modal common sense fusion of textual common sense features with image features, video features, and audio features of the video to obtain multimodal common sense features; Step S33: The multimodal common sense features are decoded using a deep video common sense decoder to generate common sense description text containing the deep meaning of the video. The deep video common sense decoder is a Transformer-based decoder, comprising: a Self-attention layer, a Cross-attention layer, and a Feedforward network layer. The Self-attention layer is used to normalize the text modal features. The Cross-attention layer is used to process cross-modal information, comparing the association between the word and the external common sense introduced by the video and GPT when generating each word. The Feedforward network layer is set after the Cross-attention layer, and searches the word library in a probabilistic form through the Feedforward layer mapping to generate the final common sense description text.

2. The video commonsense text generation method based on unstructured knowledge embedding according to claim 1, characterized in that, Step S11: Extract image features, video features, and audio features from the video, including: Image features of the video are extracted using a pre-trained ResNet152 network structure; Video features are extracted from the video using a pre-trained I3D network structure; Audio features of the video are extracted using a pre-trained SoundNet network structure.

3. The video commonsense text generation method based on unstructured knowledge embedding according to claim 2, characterized in that, Step S12: Perform cross-modal fusion of the video's image features, video features, and audio features to obtain multimodal features, including: The image features, video features, and audio features are encoded separately by multiple LSTMs and then concatenated to obtain the concatenated multimodal feature vector F′. video : F′ video =LSTM(FC(V 3D ))+LSTM(FC(I 2D ))+LSTM(FC(A 1D )) Among them, F′ video V represents the concatenated multimodal feature vector; 3D It is a video feature; I 2D It is an image feature; A 1D These are audio features; FC stands for Fully Connected Network Layer, and LSTM stands for Long Short-Term Memory Neural Network. The multimodal feature vector F′, which is concatenated, is generated using an encoder based on the Transformer architecture. video The encoding is re-fused to obtain the final multimodal features F. video The encoder based on the Transformer structure includes a set of self-attention layers and feedforward network layers. The self-attention layer is represented as follows: The concatenated multimodal feature vector F′ video Through three mapping matrices W respectively Q W K and W V This yields three input matrices Q, K, and V; d k is the normalization parameter; softmax is the normalized activation function, and T represents the transpose matrix.

4. The video commonsense text generation method based on unstructured knowledge embedding according to claim 1, characterized in that, In step S13, the training loss of the video content decoder uses the cross-entropy loss function, wherein the video content decoder loss function... Where, N cap The total length of the text description generated for the target, y t F represents the word that should be generated at position t. video For multimodal feature vectors, θ cap Here, p() represents the mapping probability, which is a model parameter. The generated words are concatenated to obtain the sentence S, which is a text description of the video content. cap The formula is as follows:

5. The video commonsense text generation method based on unstructured knowledge embedding according to claim 1, characterized in that, The training loss function L of the unstructured knowledge base GPT gpt as follows: Where N is the total sample length; token n The tokens are the words that GPT needs to generate during training. n-K θ represents the existing words input into GPT; K represents the total sentence length of the text description and prompts in the video content; gpt These are the model parameters; P() represents the mapping probability.

6. The video commonsense text generation method based on unstructured knowledge embedding according to claim 1, characterized in that, Multimodal commonsense features are obtained by fusing textual commonsense features with image, video, and audio features from the video across modalities, including: The features obtained from BERT encoding are mapped to a semantic space that can be shared with video features through feature mapping. Then, this semantic space is fused with the three-dimensional features from the video through cross-modal commonsense fusion, as shown in the following formula: in, For the fused multimodal commonsense features, FC stands for fully connected network layer, LSTM stands for Long Short-Term Memory neural network, and V... 3D It is a video feature; I 2D It is an image feature; A 1D It is an audio feature; S cms This indicates a text describing common sense, and Bert indicates that it has been encoded using the Bert model.

7. The video commonsense text generation method based on unstructured knowledge embedding according to claim 1, characterized in that, The video deep commonsense decoder uses the cross-entropy function as the training loss. In each iteration, it generates the word with the highest probability at the current position, and then generates a commonsense description text for each word. The video deep commonsense decoder loss function... as follows: Where, N cms Let y be the total length of the text describing common sense. t This indicates the word that should be generated at position t. For multimodal common sense features, S cap θ is a text description of the video content. cms Here are the decoder model parameters, and p() represents the mapping probability.

8. The video commonsense text generation method based on unstructured knowledge embedding according to claim 1, characterized in that, Total training loss It consists of two parts: the video content decoder loss function and the video deep commonsense decoder loss function. in, Let the loss function be the video content decoder. This is the loss function for the deep commonsense decoder of video.

9. A video commonsense text generation device based on unstructured knowledge embedding, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 8.

Images