Video description method and system based on text quality perception
Through text quality perception method, combined with text-label consistency and text-video similarity evaluation indicators, a description text annotation quality vector is generated, which solves the problem of difficulty in labeling quality evaluation in noise data processing in the prior art, and achieves more accurate Video description generation.
Patent Information
- Application Number
- CN202411219146.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The existing video description model has problems such as difficulty in evaluating the labeling quality and insufficient multi-dimensional evaluation in noise data processing. The CLIPScore method can only evaluate the similarity between text and image, and cannot effectively evaluate the similarity between text and video.
Using a text quality perception method, the descriptive text label quality vector is generated through text-label consistency and text-video similarity evaluation indicators (ROUGH, CIDEr, BERT, CLIP), and the minimum and maximum normalization process is performed to form a mass matrix. Generate descriptions using visual encoder and multi-layer Transformer modules.
A more comprehensive and accurate labeling quality evaluation is achieved, and it can learn from noisy data and improve the accuracy and efficiency of model training.
Smart Images

Figure CN118736468B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision and video processing, and in particular relates to a video description method and system based on text quality perception. Background Art
[0002] In the past decade, the video description task has made great progress, mainly due to the development of deep learning technology. At the same time, the collection of large-scale datasets such as MSR-VTT and MSVD has provided a large number of video-description samples for model training, which has promoted the development of the video description task. However, in order to collect these datasets, videos are sent to multiple annotators through crowdsourcing to collect subtitles describing the video. Due to the subjectivity of the annotators, noisy information is inevitably present, and noisy data will reduce the model's description generation ability.
[0003] To solve the noise problem, the latest research uses the CLIPScore method to evaluate the similarity between images and text, which is then used as a key control signal in the training process to guide the model to learn and achieve cross-modal alignment at different levels. This method has achieved significant improvements, but there are still problems. First, due to the lack of perfect labels, the quality of text annotation cannot be directly evaluated. And the annotation quality should be multi-dimensional and cannot be evaluated from only one dimension. In addition, CLIPScore only evaluates text and images, not text and video. Summary of the invention
[0004] In view of the above situation, the main purpose of the present invention is to propose a video description method and system based on text quality perception to solve the above technical problems.
[0005] The present invention proposes a video description method based on text quality perception, which comprises the following steps:
[0006] Step 1: Select all text descriptions of the same video annotation to form a pseudo-truth label for evaluating the quality of the description;
[0007] Step 2: Based on the text-label consistency, the language evaluation indicators ROUGH, CIDEr and pre-trained BERT are selected as quality evaluation indicators, and the quality of the description text annotation is evaluated using all text descriptions and pseudo-true value labels annotated by the same video to generate the first description text annotation quality;
[0008] Step 3: Based on the text-video similarity, the pre-trained CLIP is selected as the quality evaluation indicator, and all the text descriptions and videos annotated by the same video are used to evaluate the quality of the description text annotation to generate the second description text annotation quality;
[0009] Step 4: concatenate the first description text annotation quality and the second description text annotation quality corresponding to each text description to form a description text annotation quality vector set;
[0010] Step 5: Obtain a set of description text annotation quality vectors of several videos, and then perform a splicing operation to obtain a description quality matrix;
[0011] For each quality assessment index, the quality value of each description text annotation quality vector in the description quality matrix in the corresponding quality assessment index is combined into a quality vector, and then minimum and maximum normalization processing is performed to obtain a normalized quality vector;
[0012] Step 6: Embed the normalized quality vector as a quality feature, use the visual encoder to obtain the feature code of the video, and send the quality feature as a quality auxiliary signal together with the feature code into a decoder composed of a multi-layer Transformer module. Through autoregression, the quality auxiliary signal is used to assist in gradually generating a description.
[0013] The present invention further proposes a video description system based on text quality perception, wherein the system applies the video description method based on text quality perception as described above, and the system comprises:
[0014] Describe the text annotation quality assessment module, which is used to:
[0015] All text descriptions of the same video annotation are selected to form a pseudo-truth label for evaluating the quality of the description;
[0016] Based on the text-label consistency, the language evaluation indicators ROUGH, CIDEr and pre-trained BERT are selected as quality evaluation indicators. All text descriptions and pseudo-true value labels annotated in the same video are used to evaluate the quality of the description text annotation to generate the first description text annotation quality;
[0017] Based on the text-video similarity, the pre-trained CLIP is selected as the quality evaluation indicator, and all the text descriptions and videos annotated by the same video are used to evaluate the quality of the description text annotation to generate the second description text annotation quality;
[0018] splicing the first description text annotation quality and the second description text annotation quality corresponding to each text description in the first description text annotation quality and the second description text annotation quality to form a description text annotation quality vector set;
[0019] Describe the text annotation quality normalization module, which is used to:
[0020] Obtain a set of quality vectors of description text annotations of several videos, and then perform a splicing operation to obtain a description quality matrix;
[0021] For each quality assessment index, the quality value of each description text annotation quality vector in the description quality matrix in the corresponding quality assessment index is combined into a quality vector, and then minimum and maximum normalization processing is performed to obtain a normalized quality vector;
[0022] A text quality-aware description decoding module for:
[0023] The normalized quality vector is embedded as a quality feature, and the feature encoding of the video is obtained using a visual encoder. The quality feature is sent to a decoder composed of a multi-layer Transformer module together with the feature encoding as a quality auxiliary signal. The quality auxiliary signal is used to assist in gradually generating a description through an autoregressive approach.
[0024] Compared with the prior art, the present invention has the following beneficial effects:
[0025] 1. The present invention forms a vector representation describing the quality of text annotation by designing a text-label consistency evaluation method and a text-video similarity evaluation method, so that the text-label consistency and the text-video similarity can be combined at the same time, thereby making the annotation quality more comprehensive and accurate when evaluating the descriptive text.
[0026] 2. The present invention proposes a new method for normalizing description quality vectors in batch videos to ensure the existence of accurate, most comprehensive labels and least relevant labels, and after normalization, the quality values corresponding to all evaluation indicators are mapped to the same scale to speed up model training.
[0027] 3. The present invention realizes text quality-aware reasoning, achieves learning from noisy data and benefits model training.
[0028] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 A flowchart of a video description method based on text quality perception proposed by the present invention;
[0030] Figure 2 This is a schematic diagram of the structure of a video description system based on text quality perception proposed by the present invention. DETAILED DESCRIPTION
[0031] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0032] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.
[0033] See also Figure 1 This embodiment provides a video description method based on text quality perception, and the method includes the following steps:
[0034] Step 1: Select all text descriptions of the same video annotation to form a pseudo-truth label for evaluating the quality of the description;
[0035] Since the evaluation method based on text-label consistency lacks a perfect label, the present invention is based on the idea that most text descriptions annotated with the same video are semantically consistent, and utilizes all text descriptions annotated with the same video as pseudo-truth labels.
[0036] The specific steps of the above scheme are as follows:
[0037] Given an input text description , using all text descriptions from the same video annotation as pseudo-truth labels;
[0038] in, , They respectively represent the 1st, 2nd and Ith text descriptions of the same video annotation.
[0039] Step 2: Based on the text-label consistency, the language evaluation indicators ROUGH, CIDEr and pre-trained BERT are selected as quality evaluation indicators, and the quality of the description text annotation is evaluated using all text descriptions and pseudo-true value labels annotated by the same video to generate the first description text annotation quality;
[0040] The specific steps of the above scheme are as follows:
[0041] The language evaluation indicators ROUGH and CIDEr are selected as quality evaluation indicators, and the text description is calculated. and pseudo-truth labels The semantic consistency of the title is used to quantify the quality of the title to generate the description text annotation quality ,in, They represent the first i The quality value of a description text obtained by ROUGH and CIDEr quality evaluation indicators, the corresponding process has the following relationship:
[0042] ;
[0043] in, represents the quality evaluation operation of ROUGH,CIDEr, Represents a text description The description quality value calculated at the jth quality assessment indicator;
[0044] We use the pre-trained BERT as the quality evaluation indicator and first use the text embedding layer of BERT to obtain the text descriptions. Text features and pseudo-truth labels Text features ;
[0045] in, , , Represents a text description The length of the word segmentation is obtained through BERT. represents the dimension of word segmentation, represents the set of real numbers, They represent the text features generated by the 1st, 2nd and Ith text descriptions in the pseudo-truth labels respectively.
[0046] For the text features generated separately and Perform global average pooling to obtain the mean vector and ,
[0047] in, , , Respectively represent the mean vectors generated by the 1st, 2nd, and Ith text descriptions in the pseudo-truth labels;
[0048] Text features and Calculate the variance separately to get the variance vector and ; Using the variance vector and Create a diagonal matrix to get the diagonal covariance matrix and ,
[0049] in, , , Respectively represent the variance vectors generated by the 1st, 2nd, and Ith text descriptions in the pseudo-truth label; , , Respectively represent the diagonal covariance matrices generated by the 1st, 2nd, and Ith text descriptions in the pseudo-truth labels;
[0050] Using the obtained mean vector and , the diagonal covariance matrix and , the text description and pseudo-truth labels Represented as a Gaussian distribution and , the corresponding process has the following relationship:
[0051] ;
[0052] in, , denote the Gaussian distributions of the first, second, and first text descriptions in the pseudo-truth labels, respectively. represents normal distribution;
[0053] Calculate Gaussian distribution based on its representation With Gaussian distribution The Bhattacharyya distance between each element in generates a text description and pseudo-truth labels The distance vector , the corresponding process has the following relationship:
[0054] ;
[0055] ;
[0056] in, represents the determinant operation, Represents a transpose operation , represents the inverse operation, Represents a text description and pseudo-truth labels The distance vector of , , Respectively represent text descriptions The Bhattacharyya distance values calculated with the 1st, 2nd and 1st text descriptions in the pseudo-truth labels;
[0057] The distance vector generated by Calculate the mean to get the quality of the description text annotation , describing the quality of text annotation and describe the quality of the text annotation Constitute the first description text annotation quality.
[0058] Step 3: Based on the text-video similarity, the pre-trained CLIP is selected as the quality evaluation indicator, and all the text descriptions and videos annotated by the same video are used to evaluate the quality of the description text annotation to generate the second description text annotation quality;
[0059] The specific steps of the above scheme are as follows:
[0060] Uniformly sample F frames from the video, use the pre-trained CLIP visual encoder to extract the visual features of each frame and splice them to get the video features ;
[0061] in, , Respectively represent the visual features of the 1st, 2nd and Fth frames of the video sampling;
[0062] Perform mean pooling on all frames in the video feature and normalize them to obtain the global video feature , the corresponding process has the following relationship:
[0063] ;
[0064] in, represents the normalization operation, represents the global video feature, , Represents feature dimension;
[0065] The pre-trained CLIP text encoder is used to extract the global text features of the description text. ,in, ;
[0066] By calculating the global video features and global text features The similarity of the second description text annotation , the corresponding process has the following relationship:
[0067] ;
[0068] in, represents the transpose operation, Indicates the quality of the second description text annotation.
[0069] Step 4: concatenate the first description text annotation quality and the second description text annotation quality corresponding to each text description to form a description text annotation quality vector set;
[0070] The specific steps of the above scheme are as follows:
[0071] Quality of first description text annotations generated by text-label consistency evaluation method The quality of the second description text annotation generated by the text-video similarity evaluation method It performs splicing operations to obtain text description The quality vector of the description text annotation containing different scales , .
[0072] Step 5: Obtain a set of description text annotation quality vectors of several videos, and then perform a splicing operation to obtain a description quality matrix;
[0073] For each quality assessment index, the quality value of each description text annotation quality vector in the description quality matrix in the corresponding quality assessment index is combined into a quality vector, and then minimum and maximum normalization processing is performed to obtain a normalized quality vector;
[0074] The specific steps of the above scheme are as follows:
[0075] A single video contains The quality of each text description annotated in the video is evaluated, and the description quality vector set is obtained. ;
[0076] in, Respectively represent the 1st, 2nd and Ith description text annotation quality vectors of the video annotation;
[0077] Then describe the quality vector set Splice to get the description quality matrix of a single video , ;
[0078] Will include The quality vector matrix set composed of videos The quality matrix is described by splicing ,in, , Represent the description quality matrices corresponding to the 1st, 2nd and Nth videos respectively.
[0079] From the description of the quality matrix Take out the quality value of the zth index of all text descriptions and get the quality vector of the zth index ,in, , denote the zth indicator quality value of the 1st description in the 1st video, the zth indicator quality value of the 2nd description in the 1st video, and the zth indicator quality value of the Ith description in the Nth video, respectively. z represents the ROUGH, CIDEr, pre-trained BERT, and CLIP quality evaluation indicators;
[0080] The quality matrix is described by All indicators in the minimum-maximum normalization are performed to obtain the normalized mass vector matrix ,in, , Respectively represent the quality values corresponding to the ROUGH, CIDEr, pre-trained BERT and CLIP quality evaluation indicators.
[0081] Step 6: Embed the normalized quality vector as a quality feature, use the visual encoder to obtain the feature code of the video, and send the quality feature as a quality auxiliary signal together with the feature code into a decoder composed of a multi-layer Transformer module. Through autoregression, the quality auxiliary signal is used to assist in gradually generating a description.
[0082] The specific steps of the above scheme are as follows:
[0083] Using the normalized mass vector generated , which is generated into a quality embedding through the embedding layer , the embedding process has the following relationship:
[0084] ;
[0085] in, Indicates an embedding operation;
[0086] The generated words, quality embeddings, and visual features obtained by the video through the visual encoder are concatenated as the input of the model and sent to the decoder composed of a multi-layer Transformer module. The next word is predicted by using a multi-head self-attention mechanism. The corresponding process has the following relationship:
[0087] ;
[0088] ;
[0089] in, Represents visual features, represents the generated words, The text feature representing the next word, represents the embedding matrix, represents a decoder consisting of multiple layers of Transformer modules, represents the normalized exponential function, Indicates the next word, Represents the input probability distribution, word is from Sampling obtained.
[0090] It should be noted that the model adopts the form of autoregression. In the initial stage, when there are no generated words, it concatenates the concatenated text start symbol [CLS], quality embedding and visual features as input.
[0091] Please refer to Figure 2 This embodiment further provides a video description system based on text quality perception, wherein the system applies the video description method based on text quality perception as described above, and the system includes:
[0092] Describe the text annotation quality assessment module, which is used to:
[0093] All text descriptions of the same video annotation are selected to form a pseudo-truth label for evaluating the quality of the description;
[0094] Based on the text-label consistency, the language evaluation indicators ROUGH, CIDEr and pre-trained BERT are selected as quality evaluation indicators. All text descriptions and pseudo-true value labels annotated in the same video are used to evaluate the quality of the description text annotation to generate the first description text annotation quality;
[0095] Based on the text-video similarity, the pre-trained CLIP is selected as the quality evaluation indicator, and all the text descriptions and videos annotated by the same video are used to evaluate the quality of the description text annotation to generate the second description text annotation quality;
[0096] splicing the first description text annotation quality and the second description text annotation quality corresponding to each text description in the first description text annotation quality and the second description text annotation quality to form a description text annotation quality vector set;
[0097] Describe the text annotation quality normalization module, which is used to:
[0098] Obtain a set of quality vectors of description text annotations of several videos, and then perform a splicing operation to obtain a description quality matrix;
[0099] For each quality assessment index, the quality value of each description text annotation quality vector in the description quality matrix in the corresponding quality assessment index is combined into a quality vector, and then minimum and maximum normalization processing is performed to obtain a normalized quality vector;
[0100] A text quality-aware description decoding module for:
[0101] The normalized quality vector is embedded as a quality feature, and the feature encoding of the video is obtained using a visual encoder. The quality feature is sent to a decoder composed of a multi-layer Transformer module together with the feature encoding as a quality auxiliary signal. The quality auxiliary signal is used to assist in gradually generating a description through an autoregressive approach.
[0102] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0103] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A video description method based on text quality perception, characterized in that: The method comprises the following steps: Step 1: Select all text descriptions of the same video annotation to form a pseudo-truth label for evaluating the quality of the description; Step 2: Based on the text-label consistency, the language evaluation indicators ROUGH, CIDEr and pre-trained BERT are selected as quality evaluation indicators, and the quality of the description text annotation is evaluated using all text descriptions and pseudo-true value labels annotated by the same video to generate the first description text annotation quality; Step 3: Based on the text-video similarity, the pre-trained CLIP is selected as the quality evaluation indicator, and all the text descriptions and videos annotated by the same video are used to evaluate the quality of the description text annotation to generate the second description text annotation quality; Step 4: concatenate the first description text annotation quality and the second description text annotation quality corresponding to each text description to form a description text annotation quality vector set; Step 5: Obtain a set of description text annotation quality vectors of several videos, and then perform a splicing operation to obtain a description quality matrix; For each quality assessment index, the quality value of each description text annotation quality vector in the description quality matrix in the corresponding quality assessment index is combined into a quality vector, and then minimum and maximum normalization processing is performed to obtain a normalized quality vector; Step 6: embed the normalized quality vector as a quality feature, use the visual encoder to obtain the feature code of the video, and send the quality feature as a quality auxiliary signal together with the feature code into a decoder composed of a multi-layer Transformer module. Through autoregression, the quality auxiliary signal is used to assist in gradually generating a description; In step 2, based on text-label consistency, language evaluation indicators ROUGH, CIDEr and pre-trained BERT are selected as quality evaluation indicators, and all text descriptions and pseudo-true value labels of the same video annotation are used to evaluate the quality of the description text annotation to generate the first description text annotation quality method, which specifically includes the following steps: The language evaluation indicators ROUGH and CIDEr are selected as quality evaluation indicators, and the text description is calculated. and pseudo-truth labels The semantic consistency of the title is used to quantify the quality of the title to generate the description text annotation quality ,in, They represent the first i The quality value of a description text obtained by ROUGH and CIDEr quality evaluation indicators, the corresponding process has the following relationship: ; in, represents the quality evaluation operation of ROUGH,CIDEr, Represents a text description The description quality value calculated at the jth quality assessment indicator; We use the pre-trained BERT as the quality evaluation indicator and first use the BERT text embedding layer to obtain the text descriptions. Text features and pseudo-truth labels Text features ; in, , , Represents a text description The length of the word segmentation is obtained through BERT. represents the dimension of word segmentation, represents the set of real numbers, Respectively represent the text features generated by the 1st, 2nd and Ith text descriptions in the pseudo-truth label; For the text features generated separately and Perform global average pooling to obtain the mean vector and ,in, , , Respectively represent the mean vectors generated by the 1st, 2nd, and Ith text descriptions in the pseudo-truth labels; Text features and Calculate the variance separately to get the variance vector and ; Using the variance vector and Create a diagonal matrix to get the diagonal covariance matrix and ; in, , , Respectively represent the variance vectors generated by the 1st, 2nd, and Ith text descriptions in the pseudo-truth label; , , Respectively represent the diagonal covariance matrices generated by the 1st, 2nd, and Ith text descriptions in the pseudo-truth labels; Using the obtained mean vector and , the diagonal covariance matrix and , the text description and pseudo-truth labels Represented as a Gaussian distribution and , the corresponding process has the following relationship: ; in, , denote the Gaussian distributions of the first, second, and first text descriptions in the pseudo-truth labels, respectively. represents normal distribution; Calculate Gaussian distribution based on its representation With Gaussian distribution The Bhattacharyya distance between each element in generates a text description and pseudo-truth labels The distance vector , the corresponding process has the following relationship: ; ; in, represents the determinant operation, represents the transpose operation, represents the inverse operation, Represents a text description and pseudo-truth labels The distance vector of , , Respectively represent text descriptions The Bhattacharyya distance values calculated with the 1st, 2nd and 1st text descriptions in the pseudo-truth labels; The distance vector generated by Calculate the mean to get the quality of the description text annotation , describing the quality of text annotation and describe the quality of the text annotation Constitute the first description text annotation quality; In step 3, based on the text-video similarity, the pre-trained CLIP is selected as the quality evaluation indicator, and all the text descriptions and videos annotated by the same video are used to evaluate the quality of the description text annotation to generate the second description text annotation quality method, which specifically includes the following steps: Uniformly sample F frames from the video, use the pre-trained CLIP visual encoder to extract the visual features of each frame and concatenate them to get the video features ; in, , Respectively represent the visual features of the 1st, 2nd and Fth frames of the video sampling; Perform mean pooling on all frames in the video feature and normalize them to obtain the global video feature , the corresponding process has the following relationship: ; in, represents the normalization operation, represents the global video feature, , Represents feature dimension; The pre-trained CLIP text encoder is used to extract the global text features of the description text. ,in, ; By calculating the global video features and global text features , as a description of the quality of text annotation , the corresponding process has the following relationship: ; in, represents the transpose operation, Indicates the quality of the second description text annotation.
2. The video description method based on text quality perception according to claim 1 is characterized in that: In step 1, the method of selecting all text descriptions of the same video annotation to jointly form a pseudo-truth value label for evaluating the quality of the description specifically includes the following steps: Given an input text description , using all text descriptions from the same video annotation as pseudo-truth labels; in, , They respectively represent the 1st, 2nd and Ith text descriptions of the same video annotation.
3. The video description method based on text quality perception according to claim 2 is characterized in that: In step 4, the method of splicing the first description text annotation quality and the second description text annotation quality corresponding to each text description specifically includes the following steps: Quality of first description text annotations generated by text-label consistency evaluation method The quality of the second description text annotation generated by the text-video similarity evaluation method Perform splicing operations to obtain text descriptions The quality vector of the description text annotation containing different scales , .
4. The video description method based on text quality perception according to claim 3 is characterized in that: In step 5, the method of obtaining a set of description text annotation quality vectors of several videos and then performing a splicing operation specifically includes the following steps: A single video contains The quality of each text description annotated in the video is evaluated, and the description quality vector set is obtained. ; in, Respectively represent the 1st, 2nd and Ith description text annotation quality vectors of the video annotation; Then describe the quality vector set Splice to get the description quality matrix of a single video , ; Will include N The quality vector matrix set composed of videos The quality matrix is described by splicing ,in, , Represent the description quality matrices corresponding to the 1st, 2nd and Nth videos respectively.
5. The video description method based on text quality perception according to claim 4 is characterized in that: In step 5, for each quality assessment index, the quality value of each description text annotation quality vector in the description quality matrix in the corresponding quality assessment index is used to form a quality vector, and then minimum and maximum normalization processing is performed to obtain the normalized quality vector. The method specifically includes the following steps: From the description of the quality matrix Take out the quality value of the zth index of all text descriptions and get the quality vector of the zth index ; in, , denote the zth indicator quality value of the 1st description in the 1st video, the zth indicator quality value of the 2nd description in the 1st video, and the zth indicator quality value of the Ith description in the Nth video, respectively. z represents the ROUGH, CIDEr, pre-trained BERT, and CLIP quality evaluation indicators; The quality matrix is described by All indicators in the minimum-maximum normalization are performed to obtain the normalized mass vector matrix ; in, , Respectively represent the quality values corresponding to the ROUGH, CIDEr, pre-trained BERT and CLIP quality evaluation indicators.
6. The video description method based on text quality perception according to claim 5, characterized in that: In step 6, the normalized quality vector is embedded as a quality feature, a feature code of the video is obtained using a visual encoder, and the quality feature is sent to a decoder composed of a multi-layer Transformer module together with the feature code as a quality auxiliary signal. The method of using the quality auxiliary signal to assist in gradually generating a description by autoregression specifically includes the following steps: Using the normalized mass vector generated , which is generated into a quality embedding through the embedding layer , the corresponding process has the following relationship: ; in, Indicates an embedding operation; The generated words, quality embeddings, and visual features obtained by the video through the visual encoder are concatenated as the input of the model and sent to the decoder composed of a multi-layer Transformer module. The next word is predicted by using a multi-head self-attention mechanism. The corresponding process has the following relationship: ; ; in, Represents visual features, represents the generated words, The text feature representing the next word, represents the embedding matrix, represents a decoder consisting of multiple layers of Transformer modules, represents the normalized exponential function, Indicates the next word, Represents the input probability distribution, word is from Sampling obtained.
7. A video description system based on text quality perception, characterized in that: The system applies the video description method based on text quality perception according to any one of claims 1 to 6, and the system comprises: Describe the text annotation quality assessment module, which is used to: All text descriptions of the same video annotation are selected to form a pseudo-truth label for evaluating the quality of the description; Based on the text-label consistency, the language evaluation indicators ROUGH, CIDEr and pre-trained BERT are selected as quality evaluation indicators. All text descriptions and pseudo-true value labels annotated in the same video are used to evaluate the quality of the description text annotation to generate the first description text annotation quality; Based on the text-video similarity, the pre-trained CLIP is selected as the quality evaluation indicator, and all the text descriptions and videos annotated by the same video are used to evaluate the quality of the description text annotation to generate the second description text annotation quality; splicing the first description text annotation quality and the second description text annotation quality corresponding to each text description in the first description text annotation quality and the second description text annotation quality to form a description text annotation quality vector set; Describe the text annotation quality normalization module, which is used to: Obtain a set of quality vectors of description text annotations of several videos, and then perform a splicing operation to obtain a description quality matrix; For each quality assessment index, the quality value of each description text annotation quality vector in the description quality matrix in the corresponding quality assessment index is combined into a quality vector, and then minimum and maximum normalization processing is performed to obtain a normalized quality vector; A text quality-aware description decoding module for: The normalized quality vector is embedded as a quality feature, and the feature encoding of the video is obtained using a visual encoder. The quality feature is sent to a decoder composed of a multi-layer Transformer module together with the feature encoding as a quality auxiliary signal. The quality auxiliary signal is used to assist in gradually generating a description through an autoregressive approach.
Citation Information
Patent Citations
Video description generation method based on multi-concept knowledge mining and storage medium
CN114743143A
Scene text recognition method and system based on attention mechanism
CN114821569A
Video labeling method and device thereof, equipment, medium and product
CN115359402A
Unsupervised video description method based on self-constrained dynamic text features
CN116109978A
Remote sensing target classification method combined with text image generation
CN117557916A
