An Online Video Summary Generation Method Based on Cross-Scene Knowledge Transfer

Abstract

The present invention discloses an online video abstract generation method based on cross-scenario knowledge transfer. First, a multi-view video segmentation module is constructed based on a lightweight object detection method, which divides the video into multiple shot representations according to object changes; then the clip shots containing objects are input into the abstract generation module. If the model performance exceeds the performance threshold in a specific scenario, the model directly outputs the multi-view video abstract generation result generated based on deep reinforcement learning. Otherwise, the context-independent reward design module extracts abstract attributes from the dynamic scene and further analyzes and decouples the scene knowledge; subsequently, the dynamic model fine-tuning module simultaneously adopts representation learning and uses sample copies of historical data to accelerate the retraining of the model to improve the generalization ability and dynamic environment adaptation ability. Finally, the trained multi-view video abstract generation framework is used to calculate the video abstract and output the result.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and particularly relates to an online video summary generation method. Background Art

[0002] Multi-view video summary generation aims to automatically extract representative segments from videos containing multi-view information to concisely describe the original video content, and has gradually received extensive attention in the academic and industrial fields. Deep reinforcement learning uses feedback rewards to assist the supervised information of manual summaries for model learning, and achieves good results through iterative optimization, providing a new solution idea for multi-view video summary generation. However, when video data is highly dynamic, the performance of existing multi-view video summary generation methods based on deep reinforcement learning is easily affected by various factors such as weather and lighting, and shows poor performance in dynamic scenarios. In addition, in existing multi-view video summary generation research, most cameras with limited resources only serve as data acquisition devices or only perform a small amount of data processing, and a large number of collected videos usually need to be streamed to a dedicated workstation with a powerful graphics processing unit for processing, ignoring storage requirements, computational complexity, and hardware limitations, resulting in network bandwidth consumption and potentially causing privacy leakage problems. Therefore, it is extremely important for video summaries to meet the performance requirements of low response latency for context adaptation while ensuring camera hardware compatibility.

[0003] Currently, some studies have successfully applied deep reinforcement learning algorithms to summary tasks of various video types, but their applications in environments outside the training scenario are restricted. Al-Shedivat et al. proposed a simple gradient-based meta-learning algorithm, which regards the non-stationary environment as a series of stationary tasks and uses the dependence between consecutive tasks to train the agent, thus being applicable to dynamic change and adversarial scenarios. Nagabandi et al. proposed an algorithm for fast continuous online learning, which uses a deep neural network model to establish and maintain task allocation, uses the Chinese Restaurant Process (CRP) algorithm to predict the task distribution, and uses the Expectation Maximization (EM) algorithm to adjust the model to adapt to the current new task, enabling the generalization and specialization of tasks to develop naturally simultaneously. In 2020, Zhou proposed the Watch-Try-Learn model, which combines reinforcement learning, imitation learning, and meta-learning, and uses demonstrations and interactions to help the agent adapt to new and more extensive tasks faster. However, current online adaptive research on deep reinforcement learning mostly focuses on scenarios such as games and robots, and has not been applied to the field of video summary generation. How to achieve the online generation of multi-view video summaries while ensuring quality is crucial for the practicality of multi-view video summary generation. Summary of the Invention

[0004] To overcome the deficiencies of the prior art, the present invention provides an online video summary generation method based on cross-scenario knowledge transfer. First, a multi-view video segmentation module is constructed based on a lightweight object detection method. This module segments the video into multiple shot representations according to target changes. Then, the clipped shots containing targets (such as people, vehicles, etc.) are input into the summary generation module. If the model performance exceeds the performance threshold under a specific scenario, the model directly outputs the multi-view video summary generation result generated based on deep reinforcement learning. Otherwise, the context-independent reward design module extracts summary attributes from the dynamic scene. The summary attributes are extracted by a multi-scale fusion context-independent feature extraction strategy, and further analyzes and decouples the scene knowledge. The multi-scale is mainly reflected in the fusion of deep semantics, image entropy, and visual features, and these scales represent the content information and practical information of the video. Subsequently, the dynamic model fine-tuning module simultaneously adopts representation learning and uses sample copies of historical data to accelerate the retraining of the model to improve the generalization ability and dynamic environment adaptation ability. Finally, the trained multi-view video summary generation framework is used to calculate the video summary and output the result.

[0005] The technical solutions adopted by the present invention to solve its technical problems include the following steps:

[0006] Step 1: Construct a context-adaptive online reinforcement learning multi-view video summary generation framework, including three functional modules: a multi-view video segmentation module, a context-independent reward design module, and a dynamic model fine-tuning module;

[0007] Step 2: Multi-view video segmentation module;

[0008] Use the Faster-RCNN algorithm to perform object detection on the real-time video obtained by the multi-view camera based on the Raspberry Pi. After the object detection process, the clipped images containing the objects are used as the input for the subsequent modules;

[0009] Step 3: Context-independent reward design module;

[0010] Step ¾1: Deep semantic feature extraction; Represent the original video as a set of deep features in the semantic space and extract them using contrastive learning;

[0011] Extract the features O of the original video from the backbone of the object detection model q and input O q into the encoder to obtain the video representation

[0012] For semantic information, input the content description O of the video frame k into the momentum encoder to extract the jump-thought representation, that is

[0013] The loss of contrastive learning is expressed as:

[0014] L q =c q d(F q ,F k )+(1-c q )max(0,α-d(F q ,F k ))

[0015] where d(F q ,F k ) represents feature F q and F k The Euclidean distance, c q =1 means d(F q ,F k ) is a positive description, otherwise it is 0; α is used to penalize video-description pairs with irrelevant descriptions;

[0016] Step 3-2: Calculate image entropy;

[0017] The image is represented in one dimension to represent the aggregation features of the image grayscale value. The specific calculation of the image entropy is as follows:

[0018]

[0019] where p i Indicates the proportion of pixels with gray value i, 0≤i≤255 in the image;

[0020] Step 3-3: Visual feature extraction;

[0021] Visual features are extracted based on color and visual complexity, and the visual features of the image are measured in the following ways:

[0022]

[0023] in rg=R+G,yb=1 / 2(R+G)-B, R, G, B represent red, green, and blue channels respectively; σ and μ are standard deviation and mean respectively;

[0024] Visual complexity F Complex Obtained from the size ratio of the image before and after compression;

[0025] The visual features are linearly weighted fusion of color and complexity;

[0026] Step 3-4: Integrate reward design;

[0027] Integrate visual features into the model’s reward mechanism; the semantic reward is used to evaluate the semantic diversity of the generated summary by measuring the semantic differences between selected frames in the feature space;

[0028] Suppose the set of candidate key frames is Calculate R sem as the average of pairwise differences between candidate key frames, expressed as:

[0029]

[0030] where represents the semantic feature of candidate key frame t, represents the semantic feature of candidate key frame t';

[0031] The entropy reward normalizes the picture entropy feature of the video segment so that the result value is mapped between [0, 1]. R entr is the average entropy of candidate key frames, expressed as:

[0032]

[0033] The visual feature reward R vis is the average visual feature score of candidate key frames calculated after normalization;

[0034] Finally, the reward for video summary generation is expressed as:

[0035] R = αR sem + βR entr + γR vis

[0036] where α, β, and γ are importance weights;

[0037] Step 4: Dynamic model fine-tuning module;

[0038] When the dynamic model fine-tuning module receives a trigger signal from the segmentation module in Step 1, i.e., the model performance is lower than the performance threshold under the specified scenario, indicating that the situation captured by the camera has changed, the multi-view video summary generation framework starts retraining;

[0039] Use representation learning to summarize images by encoding them into dense vector representations suitable for RL, thereby reducing the size of the model state space and accelerating model retraining; attach an additional set of copies of unique samples obtained during training to the experience replay; the unique samples refer to samples with rewards higher than the set high threshold or lower than the set low threshold;

[0040] Step 4: Generate video summary;

[0041] Use the multi-view video summary generation framework trained in Step 3 to calculate the video summary and output the result.

[0042] Preferably, the model parameter θ used to extract video frame semantic information in Step 3-1 kFixed and unchanged, the gradient does not backpropagate through this layer during the contrastive learning process; It is updated using a momentum target encoding method. At the beginning of training, f q = f k As f q is updated, according to θ k = mθ k + (1 - m)θ q to update the parameters of f k .

[0043] Preferably, the method of image compression uses the JPEG compression algorithm.

[0044] The beneficial effects of the present invention are as follows:

[0045] The method of the present invention further analyzes and decouples scene knowledge through a context - independent feature extraction strategy of multi - scale fusion, achieving fast retraining in new scenes. Among them, multi - scale is mainly reflected in the fusion of deep semantics, entropy, and visual features. In addition, based on representation learning and replay buffer, the model convergence speed is improved. A large number of experiments show that the performance of the present invention is better than existing methods, with the F1 score reaching 92% and the model convergence time reduced by 20%. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a conceptual diagram of the method of the present invention.

[0047] Figure 2 and is an architecture diagram of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The present invention will be further described below in conjunction with the drawings and embodiments.

[0049] The present invention proposes an online video summary generation method based on cross - scene knowledge transfer, using the following principle: Although real - world scenes are dynamically diverse, domain similarity can be explored between different scenes to draw on relevant domain knowledge to assist in the training of the current domain model. On this basis, when the model running context changes, a pre - constructed context - independent reward design method is used to extract reward features in the new scenario, and the dynamic model fine - tuning module uses the extracted features for online retraining of the model, achieving cross - scene adaptation and accelerated convergence of retraining for video summary generation, thereby improving the generalization ability and online generation ability of the video summary model in new scenarios.

[0050] As Figure 1 [[ID=4-eight]]and Figure 2 shown, an online video summary generation method based on cross - scene knowledge transfer includes the following steps:

[0051] Step 1: Segment multi - view video shots

[0052] Since most of the monitored events of interest are usually related to human activities, the present invention uses object detection to segment videos. Specifically, the Faster-RCNN algorithm is used to perform object detection on the real-time video obtained by a multi-view camera based on a Raspberry Pi. After the object detection process, the clip shots containing the objects will be used as the input for the subsequent modules. At the same time, in order to be compatible with and adapt to camera platforms with limited resources (computing, storage), the AdaDeep algorithm is used to compress the CNN model to reduce the complexity of the model and realize the deployment and application of a lightweight object recognition model on resource-constrained platforms.

[0053] Step 2: Extract context-independent features with multi-scale fusion

[0054] In order to extract context-independent features from all angles and at the same time focus on the characteristics that key frames should possess, the present invention proposes a context-independent feature extraction strategy with multi-scale fusion to further analyze and decouple scene knowledge. The multi-scale is mainly reflected in the fusion of deep semantics, image entropy, and visual features, and these scales represent the content information and practical information of the video. The specific implementation is as follows:

[0055] Deep semantic feature extraction. Specifically, the original video is represented as a set of deep features in the semantic space, and contrastive learning is used for extraction. The input feature O q of the feature extraction is the original video feature extracted from the backbone of the object detection model, and it is input into the encoder to obtain the video representation For semantic information, the present invention inputs the content description of the video frame into the momentum encoder to extract the leapfrog thinking representation, that is The loss of contrastive learning can be expressed as:

[0056] L q = c q d(F q , F k ) + (1 - c q ) max(0, α - d(F q , F k ))

[0057] where d(F q , F k ) represents the Euclidean distance between the features F q and F k , c q = 1 means d(F q , F k) is a positive description, otherwise it is 0; α is used to penalize video-description pairs with irrelevant descriptions. Similar to instance discrimination in image settings, positive descriptions are accurate descriptions of video frames, while negative descriptions come from other video frames. The present invention mainly relies on randomly generated data augmentation, that is, randomly extracting video frame descriptions of irrelevant video segments. It should be noted that θ k is fixed and the gradient does not backpropagate through this layer during the contrast learning process, similar to the target network in DQN. is updated using a momentum target encoding method. At the beginning of training, f q = f k , as f q is updated, according to θ k = mθ k +(1 - m)θ q to update the parameters of f k .

[0058] Image entropy calculation. The present invention adopts one-dimensional image entropy, which can represent the aggregation characteristics of image gray values. Although two-dimensional entropy can reflect the characteristic quantity of the spatial characteristics of gray distribution, the calculation of two-dimensional entropy will affect the real-time performance of abstract generation, especially when executed on resource-constrained devices. The specific calculation of image entropy is as follows:

[0059]

[0060] where p i represents the proportion of pixels with gray value i (0 ≤ i ≤ 255) in the image.

[0061] Visual feature extraction. The present invention extracts visual features based on both color and visual complexity, and measures the visual features of the image in the following way:

[0062]

[0063] where rg = R(red)+(green). The image complexity F Complex is obtained from the ratio of the sizes of the image before and after compression, and the compression method adopts the JPEG compression algorithm. The visual feature is finally a linear weighted fusion of color and complexity. The visual feature is finally a linear weighted fusion of color and complexity.

[0064] Fusion reward design. In order to effectively learn the features of the abstract from the extracted context-independent features and constrain the generated abstract based on the extracted features, these features are fused into the reward mechanism of the model. The semantic reward evaluates the semantic diversity degree of the generated abstract by measuring the semantic difference between the selected frames in the feature space. Suppose the set of candidate key frames is Let R semCalculated as the average of pairwise differences between candidate key frames, which can be expressed as:

[0065]

[0066] The entropy reward normalizes the picture entropy features of the video segment so that the result value is mapped between [0,1], where R entr is the average entropy of the candidate key frames and can be expressed as:

[0067]

[0068] The visual feature reward, similar to the entropy reward, the visual feature reward R vis also calculates the average visual feature score of the candidate key frames after normalization. Finally, the reward for video summary generation can be expressed as:

[0069] R = αR sem + βR entr + γR vis

[0070] where α, β, and γ are importance weights, which can be modified according to the user's preferences to generate personalized video summaries.

[0071] Step 3: Determine whether dynamic fine-tuning of the model is required

[0072] If the model performance exceeds the performance threshold under a specific scenario, execute Step 5, and the model directly outputs the multi-view video summary generation result generated based on reinforcement learning. Otherwise, it indicates that the situation captured by the camera has changed, and the model performance will decrease due to the significant change in the data distribution. Therefore, execute Step 4, and the model will initiate retraining of the model to improve the generalization ability and dynamic environment adaptation ability.

[0073] Step 4: Dynamically fine-tune the summary generation model

[0074] Extract the context-independent features of summary multi-scale fusion from the dynamic scenario using Step 2. Subsequently, the dynamic model fine-tuning module performs retraining and updates the model to adapt to environmental changes.

[0075] To accelerate model retraining, state representation is used to reduce the model search space, and at the same time, representation learning is adopted and sample copies of historical data are used. First, representation learning is used to summarize the image by encoding the image into a dense vector representation suitable for RL, thereby reducing the size of the model state space and accelerating model retraining. Second, a set of additional copies of unique samples (samples with higher or lower rewards obtained) during the training process are appended to the experience replay to highlight them during batch training.

[0076] Step 5: Generate video summary

[0077] Using the multi-view video summary generation model trained in step 3, calculate the video summary and output the result.

Claims

1. An online video summary generation method based on cross-scenario knowledge transfer, characterized in that Including the following steps: Step 1: Construct a context-adaptive online reinforcement learning multi-view video summary generation framework, including three functional modules: a multi-view video segmentation module, a context-independent reward design module, and a dynamic model fine-tuning module; Step 2: Multi-view video segmentation module; Use the Faster-RCNN algorithm to perform object detection on the real-time video obtained by the multi-view camera based on the Raspberry Pi. After the object detection process, the clipped images containing the objects are used as the input for the subsequent modules; Step 3: Context-independent reward design module; Step 3-1: Deep semantic feature extraction; Represent the original video as a set of deep features in the semantic space and extract them using contrastive learning; Extract the features of the original video from the backbone of the object detection model O q , and input O q into the encoder to obtain the video representation For semantic information, the content description O of the video frame k is input into the momentum encoder to extract the jump-thought representation, that is The loss of contrastive learning is expressed as: L q = c q d(F q ,F k ) + (1 - c q ) max(0, α - d(F q ,F k )) where d(F q , F k ) represents the Euclidean distance between feature F q and F k , c q = 1 indicates that d(F q , F k ) is a positive description, otherwise it is 0; α is used to penalize video-description pairs with irrelevant descriptions; Step 3-2: Picture entropy calculation; Adopt the one-dimensional representation of the image to represent the aggregation feature of the image grayscale value. The specific calculation of the picture entropy is as follows: where p i represents the proportion of pixels with a gray value of i, 0 ≤ i ≤ 255, in the image; Step 3-3: Visual feature extraction; Extract visual features based on both color and visual complexity, and measure the visual features of the image in the following way: Among them rg = R + G, yb = 1 / 2(R + G) - B, where R, G, and B represent the red, green, and blue channels respectively; σ and μ are the standard deviation and the mean respectively; Visual complexity F Complex Obtained from the ratio of the sizes of the image before and after compression; The visual feature is a linear weighted fusion of color and complexity; Step 3-4: Fusion reward design; Integrate the visual features into the reward mechanism of the model; The semantic reward evaluates the semantic diversity degree of the generated summary by measuring the semantic difference between the selected frames in the feature space; Suppose the set of candidate key frames is Let R sem be calculated as the average of the pairwise differences between candidate key frames, denoted as: Among them, represents the semantic feature of the candidate key frame t, represents the semantic feature of the candidate key frame t'; The entropy reward normalizes the picture entropy features of the video clip so that the result value is mapped between [0, 1], and R entr is the average entropy of the candidate key frames, expressed as: Visual feature reward R vis is to calculate the average visual feature score of candidate key frames after normalization; Finally, the reward for video summary generation is expressed as: R = αR sem + βR entr + γR vis where α, β, and γ are importance weights; Step 4: Dynamic model fine-tuning module; When the dynamic model fine-tuning module receives the trigger signal from the segmentation module in Step 1, that is, when the model performance is lower than the performance threshold under the specified scenario, indicating that the context captured by the camera has changed, the multi-view video summary generation framework starts retraining; Use representation learning to summarize the image by encoding the image into a dense vector representation suitable for RL, thereby reducing the size of the model state space and accelerating model retraining; Attach a set of additional copies of the unique samples obtained during the training process to the experience replay; The unique samples refer to the samples whose obtained rewards are higher than the set high threshold or lower than the set low threshold; Step 4: Generate video summary; Use the multi-view video summary generation framework trained in Step 3 to calculate the video summary and output the result.

2. The online video abstract generation method based on cross-scenario knowledge transfer according to claim 1, wherein, The model parameters θ used to extract the semantic information of video frames in step 3-1 k remain fixed, and the gradient does not backpropagate through this layer during the contrastive learning process; is updated using a momentum target encoding method. At the beginning of training, f q = f k As f q is updated, according to θ k = mθ k + (1 - m)θ q to update the parameters of f k .

3. An online video abstract generation method based on cross-scenario knowledge transfer according to claim 1, characterized in that The method of picture compression adopts the JPEG compression algorithm.

Images