A video description generation method and storage medium based on multi-concept knowledge mining
By using multi-concept knowledge mining and a Transformer-structured video description generation model, the problem of insufficient utilization of multimodal information in existing technologies is solved, and high-quality and fast video description generation is achieved.
Patent Information
- Application Number
- CN202210375021.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-04-11
AI Technical Summary
Existing video description generation methods cannot fully utilize multimodal information, resulting in semantically limited generated descriptions and slow training speeds.
We employ a multi-concept knowledge mining approach, which acquires visual features and semantic tags from videos. We then utilize a Transformer-structured video description generation model to mine video-text, video-video, and text-text knowledge. Furthermore, we optimize the parameters of the multi-head self-attention layer and word embedding layer, and combine this with a large-scale word vector model for semantic tag optimization.
The generated video descriptions are of high quality, have strong generalization ability, and are trained quickly, enabling more accurate generation of grammatically and semantically rich descriptive sentences.
Smart Images

Figure CN114743143B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video description generation, and in particular to a video description generation method and storage medium based on multi-concept knowledge mining. Background Art
[0002] As videos occupy an increasing share of the Internet, new markets and application prospects are gradually being opened up. Using computers to automatically understand, analyze and process video data has become a technical need that needs to be addressed. Video description generation, as one of the key tasks of video understanding, aims to describe what happened in a video in the form of natural language. This task has broad application prospects in the fields of early childhood education, development of assistive devices for the visually impaired, and human-computer interaction. Since it involves both computer vision and natural language processing, there are certain technical difficulties in modeling temporal video information and generating natural language description sentences. Compared with the results of single-domain or static image vision tasks, there is still room for development in video description generation tasks.
[0003] Since unimodal information cannot fully characterize the information in the video, the video description generation method based on multimodal information is the current mainstream method, such as a video description text generation method based on multimodal feature fusion disclosed in CN113806587A. Many existing methods mainly obtain prior knowledge to assist the model in generating description sentences by optimizing the processing of video features and text sequences or adding additional modal information. However, the prior knowledge extracted by such methods only focuses on a single component element in the video content, such as the subject or behavior, and cannot cover all the content in the video. Therefore, studying how to make full use of multimodal information and extract more and more accurate knowledge from it to guide the model to generate video description sentences with more accurate syntax and richer semantics is of great significance to improving the performance of video description generation tasks. Summary of the invention
[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a video description generation method and storage medium based on multi-concept knowledge mining with strong subject relevance, high semantic richness and fast training speed.
[0005] The purpose of the present invention can be achieved by the following technical solutions:
[0006] A video description generation method based on multi-concept knowledge mining, comprising: obtaining an input video to be processed, extracting visual features and semantic labels from the input video, optimizing the semantic labels, obtaining prior semantic labels, using the extracted visual features and the prior semantic labels as inputs of a video description generation model based on a Transformer structure, and obtaining corresponding description results, wherein the visual features include 2D features and 3D features;
[0007] When the video description generation model is trained, video-text knowledge, video-video knowledge and text-text knowledge are mined from training samples, and parameters of the multi-head self-attention layer and the word embedding layer in the video description generation model are optimized.
[0008] Furthermore, the semantic labels include image classification prediction labels, behavior recognition prediction labels, target detection prediction labels, and probability values corresponding to each label.
[0009] Furthermore, the extraction of visual features and semantic tags and the mining of video-text knowledge are implemented based on an external pre-trained model, and the mining of video-video knowledge and text-text knowledge is implemented based on target dataset corpus training.
[0010] Furthermore, the mining of video-video knowledge and text-text knowledge is specifically as follows:
[0011] The 2D features and 3D features in the training samples are passed through an independent encoder respectively, and the 2D feature encoding and 3D feature encoding are output and concatenated for pre-training of the video description generation task. After the training is completed, the parameters of the multi-head attention layer and the word embedding layer in each encoder are extracted;
[0012] The local semantic similarity between words in the target dataset corpus is calculated based on the parameters of the word embedding layer;
[0013] The 2D features and 3D features in the training samples are used for pre-training of the video multi-label classification task, and the loss function in the training process is constructed based on the local semantic similarity.
[0014] Furthermore, the loss function is expressed as:
[0015]
[0016] y k,i;j+1 =y k,i;j +T(sim local (k,j),0.5)-y k,i;j ×T(sim local (k,j),0.5)
[0017] T(x,t)=x×sgn(max(xt),0)
[0018] Among them, p represents the probability of each word appearing at that position, p m =max(pm,0), m is a hyperparameter that controls the contribution of negative examples, γ + and γ - are the hyperparameters that control the weights of positive and negative examples, y k,i;j+1 represents the target value of the kth label in the dictionary when the i-th video is processing the j+1th label to which the video belongs, sim local (k,j) represents the local semantic similarity.
[0019] Furthermore, the semantic tags are optimized as follows:
[0020] Based on a large word vector model, the global semantic similarity between words in the dataset corpus of all semantic tags is calculated, and the local semantic similarity between words in the target dataset corpus is calculated based on the parameters of the word embedding layer. A double word mapping is established, and the mapping value of each semantic tag is obtained. The topic relevance is calculated based on the mapping value, and the semantic tags with high topic relevance are selected as prior semantic tags by setting numerical values.
[0021] Furthermore, the large word vector model is a GloVe model.
[0022] Furthermore, the calculation formulas used for the global semantic similarity and the local semantic similarity are both:
[0023]
[0024] in, and They represent the word vector representation of two words based on the word embedding layer parameters. α and β are hyperparameters that control the distribution of semantic similarity, and they take different values when calculating global semantic similarity and local semantic similarity.
[0025] Furthermore, the extracted visual features and the prior semantic labels are used as inputs of a video description generation model based on a Transformer structure as follows:
[0026] The 2D feature and 3D feature are respectively passed through an independent encoder to obtain 2D feature encoding and 3D feature encoding, the prior semantic label sequence is passed through a word embedding layer to obtain semantic feature encoding, and the 2D feature encoding, 3D feature encoding and semantic feature encoding are concatenated and sent to the decoder for prediction.
[0027] The present invention also provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the video description generation method based on multi-concept knowledge mining as described above.
[0028] Compared with the prior art, the present invention has the following beneficial effects:
[0029] 1. High generation quality: The present invention simulates the human understanding of video, and applies different tasks based on frame-level information (2D features) and inter-frame information (3D features) to mine multi-concept prior knowledge and rich semantic labels, including video-video knowledge, video-text knowledge, text-text knowledge, and other multi-concept knowledge mining; at the same time, the present invention uses the strong synergy between knowledge to further optimize the semantic labels to make them more matching to the video. The strong prior multi-concept knowledge of the present invention can effectively improve the quality of the description sentences generated by the model.
[0030] 2. Strong generalization ability: The present invention applies a variety of pre-trained and non-pre-trained models to perform knowledge mining on training samples, and optimizes the knowledge based on a large word vector model and a pre-trained word embedding layer, which effectively alleviates the semantic deviation caused by different data set corpora. When applied to different data sets, it has stronger generalization ability.
[0031] 3. Fast training speed: The present invention uses a network based on the Transformer structure. Compared with other networks based on the LSTM structure, the present invention can efficiently perform parallel calculations and has a faster convergence speed during the training process. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is a schematic diagram of the overall structure of the present invention;
[0033] Figure 2 It is a schematic diagram of the structure of the encoder of the present invention;
[0034] Figure 3 Schematic diagram of the structure of the classifier of the present invention. DETAILED DESCRIPTION
[0035] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0036] This embodiment provides a method for generating video description based on multi-concept knowledge mining, including: obtaining an input video to be processed, extracting visual features and semantic labels from the input video, optimizing the semantic labels, obtaining prior semantic labels, and using the extracted visual features and the prior semantic labels as inputs to a video description generation model based on a Transformer structure to obtain corresponding description results, wherein the visual features include 2D features and 3D features, and the semantic labels include image classification prediction labels, behavior recognition prediction labels, target detection prediction labels, and probability values corresponding to each label. When the video description generation model is trained, video-text knowledge, video-video knowledge, and text-text knowledge are mined from training samples to optimize the parameters of the multi-head self-attention layer and the parameters of the word embedding layer in the video description generation model.
[0037] The above-mentioned video-video knowledge and text-text knowledge mining is implemented based on the training of the target dataset corpus. Since image classification, behavior recognition and target detection are mature tasks, there are multiple pre-trained models with excellent performance to choose from, that is, the extraction of visual features and semantic labels and the mining of video-text knowledge are based on external pre-trained models; while the video multi-label classification and video description generation tasks are not mature, there are fewer datasets, and the existing results have different structures and performance. It is not possible to use pre-trained models to obtain the desired knowledge, so non-pre-trained models are used, that is, the mining of video-video knowledge and text-text knowledge is based on the training of the target dataset corpus.
[0038] This method can realize the mining of multiple knowledge such as video-video knowledge, video-text knowledge, and text-text knowledge, which correspond to the parameters of the multi-head attention layer, semantic labels, and word embedding layer in the encoding end, effectively improving the quality of description sentences generated by the model.
[0039] The above video description generation principle is as follows Figure 1 The method involves the training of a video description generation model, and the specific training process includes the following steps:
[0040] S1. Visual feature extraction and video-text knowledge mining of training samples based on pre-trained models, including Figure 1 It consists of three parts: target detection, image classification and behavior recognition.
[0041] Since the ResNeXt-101 model, MoViNet model and Faster R-CNN model perform well in image classification tasks, behavior recognition tasks and target detection tasks, respectively, this embodiment uses the ResNeXt-101 model and MoViNet model as 2D extractors, 3D extractors and target detection models. Given an input video, first convert the video frequency to 12 frames / second, sample 32 video frames at equal intervals, and scale them to 224×224 size. The 2D extractor extracts 2D frame-level features and predicts the category label and probability of the image for each frame, retaining the category label with a probability of more than 0.95. Then, the target detection model is used to detect the objects and their probabilities in each frame of the 32 video frames, and retain the category labels with a probability of more than 0.5. Finally, the video is sampled from the beginning with 8 consecutive frames at intervals of 4 frames, and each frame is scaled to 320×320 size. The 3D extractor is used to extract 3D inter-frame features and predict the category label and probability of the video segment for each video frame sequence consisting of 8 frames, and the category label with a probability of more than 0.9 is retained. 2D frame-level features can represent frame-level spatial domain information, 3D inter-frame features can represent inter-frame spatiotemporal domain information, category labels are fine-grained representations of the content in the video, and probabilities represent the confidence that the labels correspond to the video.
[0042] S2. Video-video and text-text knowledge mining based on non-pretrained models.
[0043] The mining of video-video knowledge and text-text knowledge is specifically as follows:
[0044] 21) The 2D features and 3D features in the training samples are passed through an independent encoder, and the 2D feature codes and 3D feature codes are output and concatenated for pre-training of the video description generation task, such as Figure 1 As shown in the video description generation (pre-training) section, the encoder structure is as follows Figure 2 As shown in the figure, after the training is completed, the parameters of the multi-head attention layer and the word embedding layer in each encoder are extracted. The parameters of the multi-head attention layer represent the model's understanding of the visual feature processing method, and the parameters of the word embedding layer represent the parameter distribution of the model projecting each word in the corpus into the feature space.
[0045] 22) Based on the parameters of the word embedding layer, the local semantic similarity between words in the target dataset corpus is calculated. The calculation formula used for semantic similarity is:
[0046]
[0047] in, and Represent the word vector representation of two words based on the word embedding layer parameters, α and β are hyperparameters that control the distribution of semantic similarity, and take different values when calculating global semantic similarity and local semantic similarity. In this embodiment, α and β used in calculating local semantic similarity are set to -5 and 10.
[0048] 23) Using the 2D features and 3D features in the training samples to pre-train the video multi-label classification task, the loss function in the training process is constructed based on the local semantic similarity.
[0049] Video multi-label classification tasks such as Figure 1 As shown in the video multi-label classification section, the classifier structure is as follows Figure 3 As shown. First, use the NLTK library to classify and restore the description sentences corresponding to the video, and use the nouns, verbs, adjectives and adverbs in the description sentences corresponding to the video as the true value labels of the video. Then, pass the 2D features and 3D features through an independent encoder respectively, splice the output 2D feature encoding and 3D feature encoding, and input them into the classifier for training. After the training is completed, predict the semantic labels in the test video. During the training process, the loss function is defined as follows:
[0050]
[0051] y k,i;j+1 =y k,i;j +T(sim local (k,j),0.5)-y k,i;j ×T(sim local (k,j),0.5) (3)
[0052] T(x,t)=x×sgn(max(xt),0) (4)
[0053] Among them, p is expressed as a matrix, each element of which represents the probability of a certain word appearing at a certain position in the description sentence of a generated video. m =max(pm,0), m is a hyperparameter that controls the contribution of negative examples, γ + and γ - are the hyperparameters that control the weights of positive and negative examples. In this example, m is set to 0.05, γ + and γ - Set to 0 and 2 respectively. k,i;j+1 represents the target value of the kth label in the dictionary when the i-th video is processing the j+1th label to which the video belongs, sim local (k,j) represents the local semantic similarity. When j = 0, y k≠j,i;j =0 and y j,i;j= 1. Thus, n rounds of iterative calculations are performed on the n labels in the i-th video. In each round of iteration, the relevance of each word in the dictionary to the video is updated according to the semantic similarity between the labels, and this is used as the target truth value of the loss function.
[0054] S3. Optimize the semantic labels, specifically: calculate the global semantic similarity between the words in the dataset corpus to which all semantic labels belong based on a large word vector model, and calculate the local semantic similarity between the words in the target dataset corpus based on the parameters of the word embedding layer, establish a double word mapping, obtain the mapping value of each semantic label, calculate the topic relevance based on the mapping value, and select the semantic label with high topic relevance as the prior semantic label by setting the numerical value.
[0055] The specific steps to optimize semantic tags include:
[0056] 31) Use the pre-trained GloVe model to extract word vectors for all words in the dataset corpus used, and apply formula (1) to obtain the global semantic similarity, where α and β are set to -10 and 20.
[0057] 32) Based on the global semantic similarity and local similarity, a global word mapping and a local word mapping are established, whereby each label can be mapped to each word in the target dataset corpus with a value, which is the product of the probability value corresponding to the label and the global semantic similarity and the local semantic similarity, which is expressed as For each word w in the target dataset dictionary, the topic relevance of its corresponding video can be calculated by the following formula:
[0058]
[0059] Among them, p j Represents the weight value of the task to which word w belongs, W j Represents the dictionary of the corresponding task corpus. When the score exceeds 1, multiple identical tags are matched to one video according to the ratio, and the 10 tags with the highest scores are taken as the semantic tags corresponding to the video. When there are less than 10 tags, the tags with the highest scores are filled to 10.
[0060] After obtaining a usable video description generation model through the above training process, 2D features, 3D features and semantic labels are extracted for a video to be described, and the 2D features and 3D features are respectively passed through an independent encoder to obtain 2D feature encoding and 3D feature encoding; based on the above semantic label optimization method, a priori semantic label sequence is obtained, and the priori semantic label sequence is passed through a word embedding layer to obtain semantic feature encoding; then, the 2D feature encoding, 3D feature encoding and semantic feature encoding are spliced and sent to the decoder for prediction to generate the corresponding description result.
[0061] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes. In order to verify the performance of the above method, the following experiment was designed.
[0062] This method is experimentally verified on two benchmark datasets, MSVD and MSR-VTT. The MSVD dataset contains 1970 videos with an average duration of 9.65 seconds, and each video clip corresponds to an average of 41.04 description sentences with an average length of 7.12 words; the MSR-VTT dataset contains 10,000 videos with an average duration of 14.88 seconds, and each video clip corresponds to 20 description sentences with an average length of 9.28 words. For both datasets, standard training / validation / test branches are used.
[0063] The evaluation indicators used in the experiment include BLEU@4, METEOR, ROUGE-L and CIDEr, four commonly used evaluation indicators in the field of video description generation. The experimental results are shown in Table 1.
[0064] In Table 1, the features used by each method are listed, and this method is represented by MCMMT. Among them, N, IRV2, R, RX, C, I, 3DRX, MVN, A, Ca and D represent the use of NasNet, InceptionResNetV2, ResNet, ResNeXt, C3D, I3D, 3D-ResNeXt-101, MoViNet, speech features, category labels in the MSR-VTT dataset and the use of target detectors. B@4, M, R, C represent the four evaluation indicators BLEU@4, METEOR, ROUGE-L and CIDEr respectively. Since the existing video description generation methods use a variety of different feature combinations, three feature combinations are also used in the experiment for a more fair comparison. Here, the combination of InceptionResNetV2 features and C3D features is called feature combination I, the combination of ResNet101 features and 3D-ResNeXt-101 features is called feature combination II, and the combination of ResNeXt101 features and MoViNet features is called feature combination III. Among the above three feature combinations, feature combination I is the most commonly used feature combination in earlier works, feature combination II is the feature combination that tends to be used in recent works, and feature combination III is the feature combination selected by this method based on comprehensive consideration of the performance of the task and the diversity of the dataset used.
[0065] Table 1 Comparison results of this method with existing methods on MSVD and MSR-VTT datasets
[0066]
[0067]
[0068] From the data in Table 1, we can see that the MCMMT method has made significant breakthroughs on both the MSVD and MSR-VTT datasets, especially in the two evaluation indicators of METEOR and CIDEr. CIDEr is an indicator designed specifically for image / video description generation tasks. Its mechanism is to penalize words that appear more frequently but have lower value in the corpus and highlight words that appear less frequently but are more important. Therefore, the huge breakthrough of MCMMT on CIDEr proves that it can more effectively capture key information in videos.
[0069] For feature combination I, MCMMT did not show better performance. The main reason is that C3D has poor feature performance, its feature extraction model structure is relatively simple, and it is pre-trained on the Sports1M dataset that only contains sports. The feature extraction model structure of 3D features in the other two feature combinations is more ingenious, and the datasets used are Kinetics-400 and Kinetics-600 datasets that contain more categories. The models trained based on this have better generalization ability. Therefore, C3D features have low sensitivity to most videos in MSVD and MSR-VTT datasets, and it is difficult to mine knowledge from them that is comparable to the other two. In contrast, MCMMT achieved comprehensive improvements on the MSVD dataset and achieved relatively excellent results on the MSR-VTT dataset when using feature combination II; it achieved a great breakthrough on the MSVD dataset when using feature combination III: BLEU@4, METEOR, ROUGE-L, and CIDEr scores increased by 4.0%, 3.7%, 3.1%, and 22.4% respectively compared with the existing best performance, and achieved the best results on the METEOR and CIDEr evaluation indicators on the MSR-VTT dataset and the second best results on both BLEU@4 and ROUGE-L.
[0070] In order to explore the effectiveness of each module in MCMMT, this method uses the video description generation model with only visual features as input and the standard Transformer as the backbone network as the baseline model, and objectively compares it with the MCMMT method. When all experimental settings and parameter settings are consistent, the ablation experiment results are shown in Table 2.
[0071] Table 2 Comparison of module-based ablation experimental results of this method on MSVD and MSR-VTT datasets
[0072]
[0073]
[0074] Among them, using only semantic labels without semantic similarity means that the mined semantic labels only use the parts that overlap with the target dataset without mapping. According to the first and second rows of data in the same feature combination experiment, it can be concluded that strong visual features can mine rich semantic knowledge to provide guidance for the model; according to the second and third rows of data in the same feature combination experiment, it can be concluded that the addition of semantic similarity also brings all-round improvement to the model, which strongly proves that the effect of semantic similarity on semantic knowledge optimization is very significant; according to the third and fourth rows of data in the same feature combination experiment, it can be concluded that the initialization parameters obtained by the pre-training model also give the model clearer guidance, avoiding falling into local saddle points during training. In addition, the results of semantic similarity optimization of the loss function of multi-label classification tasks are shown in Table 3.
[0075] Table 3 Comparison of results before and after optimization of loss function for video multi-label classification task
[0076]
[0077]
[0078] Among them, the acceptance threshold of the label is 0.8; F1 represents the harmonic mean of precision and recall, which is calculated using the standard 0-1 true value; S superimposes semantic similarity on the basis of the F1 indicator, that is, it is calculated using the optimized target true value; N represents the number of semantic labels finally mined. It can be seen that the introduction of semantic similarity effectively improves the mAP value of the result, that is, it improves the global accuracy of the model. Its improvement in N means that this improved method can mine more semantic labels from the video and avoid the situation where there are fewer labels and the model learning is insufficient. Due to the increase in the number of mined semantic labels, its F1 value decreases slightly, while the S value increases significantly, which means that the semantic labels mined after optimization are more consistent with the content of the video on the basis of considering the correlation between the labels, proving the effectiveness of this method for video multi-label classification tasks.
[0079] Finally, in order to explore the contribution of each task to semantic mining in this method, this method also conducted a set of experiments for data analysis. The results are shown in Table 4, where "MLC", "OD", "AR", and "IC" represent the video multi-label classification task, the target detection task, the behavior recognition task, and the image classification task, respectively. The "Recall" indicator represents the recall rate of semantic vocabulary in the generated description sentence and the reference description sentence. The data in the table show that the video multi-label classification task contributes the most to semantic knowledge, followed by the target detection task, while the image classification task contributes very little. The reason for this may be that the video multi-label classification model has the same data set and relatively similar task objectives as the target task. The data set used by the target detection task has a wider range of label classifications and covers more vocabulary in the data set used by the image classification task. Comparing the data in the first and third rows, when the semantic label is introduced to give the model guidance of prior knowledge, the model first improves the syntax of the generated description sentence, rather than the semantic aspect that is usually intuitively believed. The reason for this may be that the semantic knowledge gives the model a weak anchor point, allowing the model to generate more grammatically fluent description sentences around this anchor point. After that, stronger semantic knowledge guides the model to generate more semantically rich description sentences.
[0080] Table 4 Comparison of task-based ablation experimental results of this method using feature combination III on the MSR-VTT dataset
[0081]
[0082]
[0083] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0084] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A video description generation method based on multi-concept knowledge mining, characterized in that: include: Acquire an input video to be processed, extract visual features and semantic labels from the input video, optimize the semantic labels, obtain a priori semantic labels, use the extracted visual features and the a priori semantic labels as inputs of a video description generation model based on a Transformer structure, and obtain corresponding description results, wherein the visual features include 2D features and 3D features; When the video description generation model is trained, video-text knowledge, video-video knowledge and text-text knowledge are mined from training samples to optimize the parameters of the multi-head self-attention layer and the parameters of the word embedding layer in the video description generation model; The extraction of the visual features and semantic labels and the mining of the video-text knowledge are implemented based on an external pre-trained model, and the mining of the video-video knowledge and text-text knowledge is implemented based on the training of the target data set corpus; The mining of video-video knowledge and text-text knowledge is specifically as follows: The 2D features and 3D features in the training samples are passed through an independent encoder respectively, and the 2D feature encoding and 3D feature encoding are output and concatenated for pre-training of the video description generation task. After the training is completed, the parameters of the multi-head attention layer and the word embedding layer in each encoder are extracted; The local semantic similarity between words in the target dataset corpus is calculated based on the parameters of the word embedding layer; Pre-training the video multi-label classification task using the 2D features and 3D features in the training samples, wherein the loss function in the training process is constructed based on the local semantic similarity; The loss function is expressed as: , , , in, represents the probability of each word appearing in the corresponding position, , is a hyperparameter that controls the contribution of negative examples. and are the hyperparameters that control the weights of positive and negative examples, Representative Videos in the video belongs to When processing a tag, the first The target value of the label, Represents local semantic similarity.
2. The video description generation method based on multi-concept knowledge mining according to claim 1 is characterized in that: The semantic labels include image classification prediction labels, behavior recognition prediction labels, target detection prediction labels, and probability values corresponding to each label.
3. The video description generation method based on multi-concept knowledge mining according to claim 1 is characterized in that: The optimization of the semantic tag is specifically as follows: Based on a large word vector model, the global semantic similarity between words in the dataset corpus of all semantic tags is calculated, and the local semantic similarity between words in the target dataset corpus is calculated based on the parameters of the word embedding layer. A double word mapping is established, and the mapping value of each semantic tag is obtained. The topic relevance is calculated based on the mapping value, and the semantic tags with high topic relevance are selected as prior semantic tags by setting numerical values.
4. The video description generation method based on multi-concept knowledge mining according to claim 3 is characterized in that: The large word vector model is a GloVe model.
5. The video description generation method based on multi-concept knowledge mining according to claim 3 is characterized in that: The calculation formulas used for the global semantic similarity and the local semantic similarity are: , in, and Represent the word vector representation of two words based on the word embedding layer parameters, and It is a hyperparameter that controls the distribution of semantic similarity, and takes different values when calculating global semantic similarity and local semantic similarity.
6. The video description generation method based on multi-concept knowledge mining according to claim 1 is characterized in that: The extracted visual features and the prior semantic labels are used as inputs of the video description generation model based on the Transformer structure: The 2D feature and 3D feature are respectively passed through an independent encoder to obtain 2D feature encoding and 3D feature encoding, the prior semantic label sequence is passed through a word embedding layer to obtain semantic feature encoding, and the 2D feature encoding, 3D feature encoding and semantic feature encoding are concatenated and sent to the decoder for prediction.
7. A computer-readable storage medium, characterized in that: It includes one or more programs executed by one or more processors of an electronic device, and the one or more programs include instructions for executing the video description generation method based on multi-concept knowledge mining as described in any one of claims 1-6.
