Group video description method and system based on efficient relational approximation algorithm

Abstract

The application discloses a group video description method and system based on an efficient relationship approximation algorithm and belongs to the field of video understanding. The method comprises the following steps: for a video group containing target videos and reference videos, generating shared features of the target video group and shared features of the reference video group according to a cross-video shared content aggregation module based on efficient relationship approximation; outputting supplementary information of the reference videos for the target videos according to a context refining module to obtain context features. The shared features and the context features are decoded by using a decoding network, and description texts for the target video group are output. The application can generate description texts for shared content of group videos, adopts a novel efficient relationship approximation algorithm to optimize the time complexity of a model to a linear level, and refines context features by using a contrast learning strategy, so that the model performance is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of video description, and in particular to a method and system for describing group videos based on an efficient relation approximation algorithm. Background Technology

[0002] Video description aims to understand the content of a given video clip and generate a natural language descriptive text. It is an interdisciplinary field between computer vision and natural language processing. Automatic image or video description involves combinations of multiple entities, including millions of objects, background, motion, and corresponding linguistic data. Most existing methods describe a single video clip, trained using aligned and annotated video text pairs. However, due to the exponential growth in the number of online videos, the demand for automatic description of sets of videos has increased significantly, such as automatically naming a categorized set of videos and recommending new video search keywords based on browsing history.

[0003] Existing reports have proposed an automatic description generation method based on image groups. This method uses a self-attention mechanism to calculate similarity scores between each image and all other images in the group. However, videos, as temporal signals, typically contain multiple consecutive frames. Compared to a set of images, calculating the similarity scores between a set of videos requires significantly higher computational complexity. Furthermore, automatic description generation methods for image groups neglect the contextual information between the target group and the reference group. Utilizing such information could effectively improve the accuracy of understanding the content of the target group.

[0004] In summary, existing technologies cannot effectively solve the problem of video description based on group videos, resulting in limited performance in practical applications and an inability to accurately and quickly generate natural language description text. Summary of the Invention

[0005] The purpose of this invention is to solve the problems in the prior art. It proposes a group video description method and system based on an efficient relation approximation algorithm. The method uses a group-based video feature aggregation network to identify and generate shared features of target video groups and shared features of reference video groups. It uses an efficient relation approximation algorithm to reduce computational complexity. At the same time, it considers inter-group self-supervised information and uses an inter-group contrast loss strategy to effectively improve model performance.

[0006] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0007] A method for describing group videos based on an efficient relation approximation algorithm includes the following steps:

[0008] 1) Establish a training dataset containing video groups and reference videos, along with descriptive text for common verb groups, and extract frame features from the target video group and the reference video group respectively;

[0009] 2) Establish an encoder-decoder network structure, including a decoder network and an encoder network consisting of a shared content aggregation module and a context feature refinement module;

[0010] 3) The frame features of the target video group and the frame features of the reference video group are used as inputs to the shared content aggregation module in the coding network to generate shared features of the target video group and shared features of the reference video group, respectively.

[0011] 4) The generated target video group shared features and reference video group shared features are used as input to the context feature refinement module, and inter-group self-supervised contrastive learning is introduced in the context feature refinement module to output context features and inter-group self-supervised loss.

[0012] 5) Concatenate the shared features of the target video group, the shared features of the reference video group, and the context features as input to the decoding network, generate a dictionary probability distribution, and select the word with the highest score as the output. After repeating this several times, compare the output with the text describing the common verb group to generate cross-entropy loss. Combine the inter-group self-supervised loss from step 4) to train the encoder-decoder network structure.

[0013] 6) For the video group for which a description is to be generated, extract the frame features of the target video group and the reference video group respectively, and use the trained encoder-decoder network to generate the description text.

[0014] Another objective of this invention is to propose a group video description system based on an efficient relation approximation algorithm, comprising:

[0015] The data acquisition module is used to acquire video sets containing the target video and reference video, as well as descriptive text of common verb phrases, as training datasets during the training phase; and to acquire video sets containing the target video and reference video to be described during the detection phase.

[0016] The visual feature extraction module is used to extract the frame features of the target video group and the frame features of the reference video group to obtain the visual features of each video in the video group.

[0017] The encoding-decoding network module is configured with a decoding network and an encoding network consisting of a shared content aggregation module and a context feature refining module;

[0018] The encoding network works as follows: the frame features of the target video group and the frame features of the reference video group are used as inputs to the shared content aggregation module in the encoding network to generate shared features of the target video group and shared features of the reference video group, respectively; the generated shared features of the target video group and shared features of the reference video group are used as inputs to the context feature refinement module to output context features.

[0019] The decoding network works as follows: the shared features of the target video group, the shared features of the reference video group, and the context features are concatenated as the input of the decoding network to generate a dictionary probability distribution and select the word with the highest score as the output. This process is repeated several times to generate descriptive text.

[0020] The training module is used to update the parameters of the encoder-decoder network module during the training phase by combining cross-entropy loss and inter-group self-supervised loss.

[0021] Compared to traditional video description methods, this invention addresses the problem of group video description and effectively improves its performance. The beneficial effects of this invention are:

[0022] (1) This invention extends the video description problem to group video description. Compared with traditional single video description, describing group videos has more practical applications. To address the problem of automatic description of group videos, this invention designs a group-based video feature aggregation module to identify and generate shared features among video groups, enhance keyframe features that are identical to those shared within the same group, and weaken irrelevant frame features.

[0023] (2) For long sequence modeling across videos, this invention proposes an efficient relation approximation algorithm with linear complexity, which takes and generates an approximate feature sequence from the input video frame sequence, and further calculates the semantic consistency score between each video frame and other approximate visual feature sequences.

[0024] (3) Regarding the contextual information between the target group and the reference group, this invention designs a novel context refinement module. This module receives shared features from both the target video group and the reference video group, and uses a multi-head attention mechanism to generate features containing contextual reference information. Furthermore, this invention also considers inter-group self-supervised signals. Through inter-group contrast loss, the target video group can distinguish supplementary information from irrelevant interference information in the reference video, making the obtained contextual features more accurate and improving the accuracy of the video description results. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating a group video description method based on an efficient relation approximation algorithm, as shown in an embodiment of the present invention.

[0026] Figure 2This is a schematic diagram illustrating the construction of a dataset according to an embodiment of the present invention. Detailed Implementation

[0027] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0028] like Figure 1 As shown, the present invention proposes a group video description method based on an efficient relation approximation algorithm, which includes the following steps:

[0029] Step 1: Create a group of videos containing the target video and reference videos, along with descriptive text, as a training dataset.

[0030] Step 2: Extract video frame features from the group of videos; then generate shared features of the target video group and shared features of the reference video group through a shared content aggregation module based on efficient relationship approximation.

[0031] Step 3: For the generated target video group shared features and reference video group shared features, a context feature refinement module is used to generate context features, and the context features are refined through an inter-group comparison learning loss function.

[0032] Step 4: Input the context features and video group shared features into the text decoder, decode N times to obtain the predicted descriptive text, and update the model parameters based on the descriptive text cross-entropy loss and the inter-group contrastive learning loss function.

[0033] Step 5: For the video group for which a description needs to be generated, obtain the natural language description text as the result based on the trained shared content aggregation module, context features, and text decoder.

[0034] In one specific embodiment of the present invention, the implementation process of step one is as follows:

[0035] The dataset construction process is as follows: Figure 2 As shown

[0036] 1.1) Obtain a publicly available video description dataset, which contains multiple videos and description text pairs;

[0037] 1.2) Randomly select video and descriptive text pairs, use a pre-trained parsing tree to parse the descriptive text, select verb groups from the parsing results, wherein the verb group consists of a verb and its corresponding noun; then combine videos with the same verb group to form a target video group, and the corresponding verb group is used as the descriptive text of the target video group; combine videos with descriptive texts that have the same noun or verb as the verb group to form a reference video group.

[0038] 1.3) Manually clean up meaningless data samples with too many or too few video groups. In this embodiment, a training sample is finally set up with 3 target videos, 5 reference videos and corresponding common verb group description text.

[0039] 1.4) Extract frame features. The specific frame feature extraction method is as follows: A pre-trained video feature extractor is used to extract the visual features of the video, resulting in a frame feature sequence for each video in the video group, denoted as V. Any feature of any video in the video group is denoted as... Where m is the frame number of any video in the video group, and v i fi is the frame sequence feature of the i-th video in the video group. i q It is the feature of the q-th frame of the i-th video in the video group. In this embodiment, all videos in the video group have the same number of frames.

[0040] In one specific embodiment of the present invention, the implementation process of step two is as follows:

[0041] 2.1) Using the target video frame features and reference video frame features as input to the shared content aggregation module, calculate the semantic consistency score between each frame in each video and any combination of frames in all other videos.

[0042] In traditional aggregation methods, it is necessary to consider the frame sequence features {v} of the input video group consisting of n videos. i The process involves iterating through each frame in each video and combining it with any frame from all other videos, then calculating the semantic consistency score for each combination. However, this method is extremely complex. Specifically, it involves combining video frame features. The semantic consistency score has in, The attention graph representing the semantic consistency scores among all combinations has a computational complexity of O(m). n ).

[0043] To address the excessive complexity of attention graph A, this invention proposes a shared content aggregation module based on efficient relation approximation. Firstly, an efficient relation approximation algorithm replaces the originally indivisible nonlinear softmax kernel function. The formula for the efficient relation approximation algorithm is as follows:

[0044]

[0045] in, Let ω be the expected value, μ be the mean, and σ be the variance. Let the random variable ω follow a sampling distribution with mean μ and variance σ as follows: ‖·‖ represents modulo. Let δ ′ (v iThe output of |i∈[1,n]) is This represents the approximate visual features obtained after sampling.

[0046] Based on an efficient relation approximation algorithm, this invention does not require traversing every frame in each video and any combination of frames in all other videos; that is, it does not require calculating the attention map composed of semantic consistency scores.

[0047] Based on approximate visual features Semantic weight scores can be calculated. The semantic weight score of the q-th frame of the i-th video. The calculation is as follows:

[0048]

[0049] By employing the efficient relation approximation algorithm proposed in this invention, the complexity of calculating the semantic consistency score between each frame in each video and other video frames can be reduced to O(n), that is, the complexity increases linearly with the number of videos n in the group.

[0050] 2.2) Based on the semantic consistency score, the visual features in the video group are aggregated, strengthening the video frame features shared with other videos in the group and weakening the video frame features unrelated to other videos in the group. The video feature aggregation formula is as follows:

[0051]

[0052]

[0053] Where, ψ tar To share features among target video groups, ψ ref To reference the shared features of the video group, n tar n is the number of videos in the target video group. ref This refers to the number of videos in the reference video group.

[0054] In one specific embodiment of the present invention, the implementation process of step three is as follows:

[0055] 3.1) Aggregate features ψ of the two video groups generated by the shared content aggregation module tar and ψ ref Then, a context refinement module is used to extract contextual information from the reference video group. Specifically, in the multi-head cross-attention mechanism, ψ tar Set as key, ψ ref Given query and value, the multi-head cross-attention formula is as follows:

[0056]

[0057] Among them, MHA is a multi-head attention mechanism, characterized by ψ tar Useful contextual information is extracted from the reference video through residual connections of self-attention blocks. The extracted contextual information is then... Share features ψ with the target video group tar Adding them together yields a more complete set of shared features ψ for the target video group. ‘ tar .

[0058] 3.2) For the generated contextual features, contrastive learning is used to introduce inter-group self-supervised signals to refine the contextual features, specifically:

[0059] Considering that text typically captures most salient events in paired visions while ignoring background features, and that videos contain a significant amount of noise, the difference between the video and the corresponding descriptive text becomes even greater. Cross-attention ignores the self-supervised signal between groups, failing to guarantee that the learned features will have ideal accuracy. If we directly obtain... The semantic features it contains will not be able to supplement the shared features of the target video group.

[0060] Maximize contextual features by introducing inter-group self-supervised signals. Shared features ψ with target group videos tar The mutual information between them can filter out interfering information in the reference video that is irrelevant to the target group video. Specifically, we will use contextual features... Each video feature pair in the target video group is used as a positive sample. Each video feature in the reference video is used as a negative sample. Target video group shared features ψ tar As a real-world example, the formula for inter-group self-supervised contrastive learning is as follows:

[0061]

[0062] in, For the inter-group self-supervised loss, s(p,q)=p T q / ‖p‖‖q‖ is a dot product operation with L2 regularization, and τ is the temperature coefficient.

[0063] In one specific embodiment of the present invention, step four is performed as follows:

[0064] 4.1) Based on the complete target video group shared features ψ generated by the shared content aggregation module and the context refinement module. ′ tar Shared feature ψ with reference video group refThe input is concatenated to form the input x = [ψ] of the decoder. ′ tar ,ψ ref ].

[0065] 4.2) x is used as the input to the decoding neural network for N-step decoding. During the decoding process, the cross-attention module in the decoder is masked, and the hidden layer sequence calculated in the decoder for the preceding symbols in the descriptive text is stored in the buffer. The data in the buffer is then introduced into the decoding process of the subsequent symbols. This decoding is repeated N times. The decoding formula for the t-th decoding is as follows:

[0066]

[0067] in, z is the description text generated in the previous decoding step. t This is the hidden layer state. The decoded description text is derived from the hidden layer state z. t The softmax function generates the probability distribution of the corresponding dictionary, and the word with the highest probability is selected as the result.

[0068]

[0069] Here, FC refers to the linear transformation layer.

[0070] 4.3) Compare the decoder output with the actual description text to calculate the cross-entropy loss function. The formula for the loss function is as follows:

[0071]

[0072] Where T is the maximum length of the description text, and d t For the t-th word in the text that describes a real communal verb phrase, d 1:T-1 This represents a sequence of real common verb phrases. Finally, a hyperparameter λ is introduced to balance the text cross-entropy loss and the inter-group self-supervised loss. The loss function formula for the task is as follows:

[0073]

[0074] By optimizing the loss function The encoding neural network, consisting of a shared content aggregation module and a context refinement module, and the decoding neural network, consisting of a text decoder, are trained until a well-trained encoding-decoding network model is obtained.

[0075] In one specific embodiment of the present invention, step five involves obtaining a target video group and a reference video group based on the actual application scenario. For example, in a video search scenario, videos that the user has clicked can be grouped into a target video group, and videos that the user has not clicked can be used as a reference video group. A pre-trained video feature extractor is used to extract the visual features of the two video groups respectively, and then a natural language description text for the input video group is generated based on the trained encoder-decoder network model.

[0076] In one specific embodiment of the present invention, a group video description system based on an efficient relation approximation algorithm is also proposed, comprising:

[0077] The data acquisition module is used to acquire video sets containing the target video and reference video, as well as descriptive text of common verb phrases, as training datasets during the training phase; and to acquire video sets containing the target video and reference video to be described during the detection phase.

[0078] The visual feature extraction module is used to extract the frame features of the target video group and the frame features of the reference video group to obtain the visual features of each video in the video group.

[0079] The encoding-decoding network module is configured with a decoding network and an encoding network consisting of a shared content aggregation module and a context feature refining module;

[0080] The encoding network works as follows: the frame features of the target video group and the frame features of the reference video group are used as inputs to the shared content aggregation module in the encoding network to generate shared features of the target video group and shared features of the reference video group, respectively; the generated shared features of the target video group and shared features of the reference video group are used as inputs to the context feature refinement module to output context features.

[0081] The decoding network works as follows: the shared features of the target video group, the shared features of the reference video group, and the context features are concatenated as the input of the decoding network to generate a dictionary probability distribution and select the word with the highest score as the output. This process is repeated several times to generate descriptive text.

[0082] The training module is used to update the parameters of the encoder-decoder network module during the training phase by combining cross-entropy loss and inter-group self-supervised loss.

[0083] In one specific embodiment of the present invention, the encoding-decoding network module includes:

[0084] The shared content aggregation module is used to sample the visual features of each video in the input video group using an efficient relation approximation algorithm to obtain approximate visual features; calculate semantic weight scores based on the approximate visual features; and aggregate the visual features in the input video group based on the semantic weight scores to generate shared features of the target video group and shared features of the reference video group.

[0085] The context feature refinement module is used to use the shared features of the target video group as the key in the multi-head cross-attention mechanism, the shared features of the reference video group as the query and value in the multi-head cross-attention mechanism, and the calculation result of the multi-head cross-attention mechanism as the context feature.

[0086] The decoding network is used to concatenate the shared features of the target video group, the shared features of the reference video group, and the context features, and then decode them to generate descriptive text.

[0087] In one specific embodiment of the present invention, the training module includes:

[0088] The inter-group self-supervised loss unit is used to refine contextual features, filter out information that is not related to the shared features of the target video group, and calculate the inter-group self-supervised loss.

[0089] The cross-entropy loss unit is used to compare the output of the encoding network with the text describing the common verb phrases to generate cross-entropy loss.

[0090] The implementation of each module mentioned above can be found in the description of the methods section, and will not be repeated here.

[0091] In the specific embodiments provided in this application, it should be understood that the system embodiments described above are merely illustrative. For example, the context feature extraction module can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the connections between the modules shown or discussed may be communication connections through some interfaces, which may be electrical or other forms.

[0092] The above method will be applied to the following embodiments to demonstrate the technical effects of the present invention. The specific steps in the embodiments will not be repeated.

[0093] This invention reconstructs the group video description datasets G-ActivityCaption and G-YouCook2 based on the ActivityCaption and YouCook2 datasets using a syntax tree parsing method:

[0094] The G-ActivityCaption dataset contains 4035 video groups with an average video length of 40.3 seconds; the dataset has 2421, 403, and 1211 video groups used for training, validation, and testing, respectively.

[0095] The G-YouCook2 dataset contains 1763 video groups, with an average video length of approximately 16.1 seconds; the dataset contains 1058 sentence-time pairs for training, 176 for validation, and 529 for testing.

[0096] Specifically, while the G-ActivityCaption dataset excels in video diversity, its descriptive text tends to be lengthy and contains significant noise. In practical applications, descriptive text for group video descriptions should be concise and compact, while G-YouCook2 offers more granular descriptions, and many descriptive texts in this dataset are more focused on describing a single, short event.

[0097] Regarding testing and evaluation criteria, this invention follows widely used standards, adopting BLEU, METEOR, ROUGE-L, and CIDEr as the standards for G-ActivityCaption and G-YouCook2.

[0098] The implementation details are as follows:

[0099] For videos, this embodiment extracts 10 frames from each video. A ResNet-101 model pre-trained on ImageNet is used to extract video frame features.

[0100] For text descriptions, this embodiment sets the maximum length to 10. Punctuation marks are removed from each description text and converted to lowercase English letters. A [sos] tag is added at the beginning of each description and a [eos] tag is added at the end.

[0101] Tables 1 and 2 present the experimental results of this invention on the G-ActivityCaption and G-YouCook2 datasets. The Average method directly averages the video frame features instead of assigning weights based on semantic consistency scores; the Traversal method obtains the semantic consistency score by iterating through every frame in each video and combining it with any frame from all other videos. The method proposed in this invention is abbreviated as ERA.

[0102] Table 1 shows the experimental results on the G-ActivityCaption dataset.

[0103]

[0104] Table 2 shows the experimental results on the G-YouCook2 dataset.

[0105]

[0106] However, as can be seen from Tables 1 and 2, this invention uses a group-based video feature aggregation module to identify and generate shared features between video groups. This module enhances keyframe features identical to those shared within the same group while weakening irrelevant frame features. Simultaneously, it considers contextual refinement information and inter-group self-supervised contrastive learning. By receiving shared features from both the target and reference video groups and employing a multi-head attention mechanism to generate features containing contextual reference information, and using inter-group contrastive loss to distinguish supplementary information from irrelevant interference information in the reference video, this invention significantly outperforms the Average method in group video description performance. Specifically, ERA achieves a 10.2% CIDEr improvement on G-YouCook2 and an 8.5% CIDEr improvement on G-ActivityCaption. Furthermore, compared to the Traversal method, which serves as the theoretical upper bound for group video description performance, this model also achieves competitive results, reaching 179.4 and 180.8 on G-YouCook2, respectively. It is worth mentioning that ERA achieved these competitive results with only 0.4% of the traversal training time, which undoubtedly demonstrates the superior performance of the present invention.

[0107] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for group video description based on efficient relational approximation algorithm, characterized in that, The method comprises the following steps: 1) establishing a video group containing target videos and reference videos and a common verb group description text as a training data set, and extracting frame features of the target video group and frame features of the reference video group respectively; 2) establishing an encoding-decoding network structure, including a decoding network and an encoding network composed of a shared content aggregation module and a context feature refinement module; 3) inputting the frame features of the target video group and the frame features of the reference video group into the shared content aggregation module in the encoding network respectively to generate shared features of the target video group and shared features of the reference video group respectively; 4) inputting the generated shared features of the target video group and the shared features of the reference video group into the context feature refinement module, introducing inter-group self-supervised contrast learning in the context feature refinement module, and outputting context features and inter-group self-supervised loss; 5) concatenating the shared features of the target video group, the shared features of the reference video group and the context features to input into the decoding network, generating a dictionary probability distribution and selecting the highest scoring word as the output, repeating several times, comparing the output result with the common verb group description text to generate a cross-entropy loss, and training the encoding-decoding network structure in combination with the inter-group self-supervised loss of step 4); 6) for a video group to be described, extracting frame features of the target video group and frame features of the reference video group, and generating a description text by using the trained encoding-decoding network.

2. The method of claim 1, wherein the method is based on a high-efficiency relational approximation algorithm. The decoding network adopts the decoder in the Transformer.

3. The method of claim 1, wherein the method is based on a high-efficiency relational approximation algorithm. Step 1) is specifically: 1.1) obtaining a video description data set, wherein the video description data set contains a plurality of video and description text pairs; 1.2) randomly selecting a video and description text pair, parsing the description text using a pre-trained syntax parsing tree, selecting a verb group in the parsing result, wherein the verb group is composed of a verb and corresponding noun groups; then combining videos with the same verb group into a target video group, and the corresponding common verb group is taken as the description text of the target video group; combining the video groups of the description texts with the same noun or verb as the reference video group; 1.3) reducing the number of videos in the target video group and the reference video group obtained in step 1.2) to a preset value, and combining the final target video group, reference video group and common verb group description text into a training sample; 1.4) Extract the visual features of each video in the video group using the pre-trained video feature extractor, and obtain the frame feature sequence of each video in the video group. The visual features of any video in the video group are denoted as wherein, is the number of frames of any video in the video group, is the frame sequence feature of the i-th video in the video group, is the q-th frame feature of the i-th video in the video group.

4. The method of claim 1, wherein the method is based on a high-efficiency relational approximation algorithm. The calculation method of the shared content aggregation module in the encoding network is: 3.1) sampling visual features of each video in the input video set using a high-efficiency relational approximation algorithm obtaining approximate visual features , and a calculation formula of the high-efficiency relational approximation algorithm is: ； wherein, is desired, is a random variable, is a mean, is a variance, let X be a random variable obeys a mean , a variance of a sampling distribution is , denotes a modulo operation, denotes the number of videos in the input video group; is a high efficient relationship approximation algorithm; 3.2) According to the approximate visual features The semantic weight score is calculated according to the following formula: ； wherein, denotes a semantic weight score of the q-th frame of the i-th video in the input video group, denotes a 1 -vector; denotes an approximate visual feature of the j-th video; 3.3) aggregating the visual features in the input video group according to the semantic weight score, and the aggregation formula is: ； ； wherein, share features for a target video group, share features for a reference video group, is the number of videos in the target video group, is the number of videos in the reference video group, is the q-th frame feature of the i-th video in the video group.

5. The method of claim 1, wherein the method is based on a high-efficiency relational approximation algorithm. The context feature refinement module adopts a multi-head cross-attention mechanism, takes the shared features of the target video group as the key in the multi-head cross-attention mechanism, takes the shared features of the reference video group as the query and value in the multi-head cross-attention mechanism, and takes the calculation result of the multi-head cross-attention mechanism as the context features.

6. The method of claim 1, wherein the method is based on a high-efficiency relational approximation algorithm. The loss function formula of the inter-group self-supervised contrast learning is: ； wherein, is the inter-group self-supervised loss, is a dot product operation with L2 regularization, is a temperature coefficient, is a positive sample obtained by concatenating the context feature and the visual feature of each video in the target video group, is a negative sample composed of the visual feature of each video in the reference video group, is a shared feature of the target video group, is the number of videos in the reference video group.

7. The method of claim 1, wherein the method is based on a high-efficiency relational approximation algorithm. The cross-entropy loss function formula is: ； wherein, is a cross-entropy loss, T is a set maximum length of the description text, is the t-th word in the real common verb group description text, is the real common verb group description text sequence, is the description text generated at the t-th decoding step.

8. A group video description system based on an efficient relational approximation algorithm, characterized by, It comprises: A data acquisition module, when the system is in a training phase, it is used for acquiring a video group containing target videos and reference videos and a common verb group description text as a training data set; When the system is in the detection stage, it is used to obtain a video group to be described, which contains target videos and reference videos; a visual feature extraction module, configured to extract frame features of the target video group and frame features of the reference video group, to obtain visual features of each video in the video group; an encoding-decoding network module, configured with a decoding network and an encoding network composed of a shared content aggregation module and a context feature refinement module; the working method of the encoding network is to input the frame features of the target video group and the frame features of the reference video group into the shared content aggregation module of the encoding network respectively, to generate shared features of the target video group and shared features of the reference video group respectively; input the generated shared features of the target video group and the shared features of the reference video group into the context feature refinement module as inputs, and output context features; the working method of the decoding network is to splice the shared features of the target video group, the shared features of the reference video group and the context features as inputs of the decoding network, to generate a dictionary probability distribution and select the word with the highest score as output, and repeat several times to generate a description text; a training module, configured to update the parameters of the encoding-decoding network module in the training stage by combining cross-entropy loss and inter-group self-supervised loss.

Images