Video analysis report generation method and system

Abstract

The invention relates to the technical field of data processing, in particular to a video analysis report generation method and system. The method comprises the steps of firstly obtaining a to-be-analyzed video; performing feature extraction on the to-be-analyzed video by using a preset feature extraction model to obtain a multi-modal feature sequence; inputting the multi-modal feature sequence into a preset video analysis model to obtain a prediction result; and finally generating an analysis report based on a prediction result. According to the method, deep fusion of multi-modal data can be realized, and the reliability of generating the analysis report is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for generating video analysis reports. Background Technology

[0002] In the digital media era, video has become a core carrier of information dissemination and user interaction. However, most current video analysis tools are limited to unimodal analysis, such as using computer vision technology to extract object and motion features from the video for analysis, or using natural language processing technology to parse the semantics of subtitle text. This unimodal analysis cannot cover the multidimensional connections within the video, leading to a semantic gap in the understanding of video content, meaning that the output analysis results deviate significantly from human cognition of the video.

[0003] To address this, existing technologies attempt to compensate for the shortcomings of single-modal analysis by integrating multimodal data. However, when performing multimodal analysis, shallow fusion methods such as simple splicing and weighting are often used, which cannot effectively capture the complex dynamic relationships between modalities. For example, simply matching the object recognition results in the image with subtitle keywords cannot capture complex interactive relationships such as the synchronicity between "image actions and voice emotions".

[0004] Therefore, how to perform deep fusion of multimodal data to improve the reliability of generated video analysis reports is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] To address the technical problem that it is difficult to capture the dynamic correlation between different modalities when fusing multimodal data through splicing or simple weighting methods, the present invention provides solutions in the following aspects.

[0006] In a first aspect, the present invention provides a video analysis report generation method, comprising: acquiring a video to be analyzed; extracting features from the video to be analyzed using a preset feature extraction model to obtain a multimodal feature sequence; inputting the multimodal feature sequence into a preset video analysis model to obtain a prediction result, specifically: inputting the multimodal feature sequence into corresponding temporal encoders to obtain a context-encoded multimodal encoding sequence; inputting the multimodal encoding sequence into a multi-layer cross-modal encoder for attention-weighted fusion to obtain a fused feature sequence; inputting the fused feature sequence into multiple parallel task prediction heads to obtain prediction results; and generating an analysis report based on the prediction results.

[0007] Beneficial effects: By employing a multi-layer cross-modal encoder to perform attention-weighted fusion of multimodal coded sequences, the video analysis model can learn the nonlinear correlation between different modal features, thereby improving the prediction accuracy of the video analysis model and thus enhancing the reliability of the generated analysis report.

[0008] Furthermore, the multimodal coding sequence is input into a multi-layer cross-modal encoder for attention-weighted fusion, including: in each layer of the cross-modal encoder, a query vector is constructed with one modal coding sequence, and key and value vectors are constructed with other modal coding sequences; cross-modal attention is calculated based on the query vector, key vector, and value vector.

[0009] Furthermore, the multimodal coding sequence includes visual feature coding sequences, audio feature coding sequences, and text feature coding sequences. If the query vector is constructed using the visual feature coding sequence, and the key vector and value vector are constructed using the audio feature coding sequence and the text feature coding sequence, then the calculation expression for cross-modal attention is:

[0010] In the formula, For cross-modal attention; The query vector is derived from a sequence of visual feature codes. The key vector is derived from the concatenation of the audio feature encoding sequence and the text feature encoding sequence; It is a value vector, derived from the concatenation of audio feature encoding sequences and text feature encoding sequences; is the scaling factor; Softmax() is the Softmax activation function.

[0011] Furthermore, the prediction results include user retention rate and the probability of interactive events; the fused feature sequence is input into multiple parallel task prediction heads to obtain prediction results, including: inputting the fused feature sequence into the first task prediction head to obtain the user retention rate at different time points; inputting the fused feature sequence into the second task prediction head to obtain the probability of each interactive event.

[0012] Furthermore, the method also includes: determining retention prediction loss, interaction event prediction loss, and explainable attribution loss; performing a weighted summation of retention prediction loss, interaction event prediction loss, and explainable attribution loss to obtain a composite loss; and using the composite loss to train and optimize the video analysis model.

[0013] Furthermore, the retention prediction loss, interaction event prediction loss, and explainable attribution loss are determined, including: calculating the retention prediction loss using the Huber loss function; calculating the interaction event prediction loss using the binary cross-entropy loss function with class weights; and calculating the explainable attribution loss using KL divergence.

[0014] Furthermore, the expression for calculating the explainable attribution loss is as follows:

[0015] In the formula, For explainable attribution loss, KL-Divergence() is the KL divergence function. Input fusion features The gradient of the prediction result output by the video analysis model. Key events marked by experts in the first The importance distribution of frames, where Softmax() is the Softmax activation function.

[0016] Furthermore, the method also includes: obtaining expert verification instructions for the analysis report; if the expert verification instructions indicate that the verification failed, obtaining the corrected data provided by the experts, and using the corrected data to iteratively optimize the weights of retention prediction loss, interactive event prediction loss, and explainable attribution loss.

[0017] Furthermore, the feature extraction models include large visual models, audio models, and language models; the multimodal feature sequences include visual feature sequences, audio feature sequences, and text feature sequences.

[0018] In a second aspect, the present invention provides a video analysis report generation system, including a processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement the video analysis report generation method described in the first aspect.

[0019] The beneficial effects of this invention are as follows: Compared with the prior art, the method of this invention can capture deep, nonlinear dynamic correlations between different modalities through a multi-layer cross-modal encoder; by incorporating interpretable attribution loss, it guides the video analysis model to learn the attribution logic of human experts, and at the same time solves the problem of the lack of interpretability of black box models such as video analysis models. Attached Figure Description

[0020] Figure 1 This is a flowchart of the video analysis report generation method in an embodiment of the present invention; Figure 2 This is a structural block diagram of the video analysis report generation system in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart of the video analysis report generation method in an embodiment of the present invention.

[0024] In a first aspect, the present invention provides a method for generating video analysis reports. Specifically, as Figure 1 As shown, the method of the present invention includes the following steps.

[0025] S1. Obtain the video to be analyzed.

[0026] Specifically, it acquires videos to be analyzed that are uploaded by users through their terminals / platforms.

[0027] S2. Use the preset feature extraction model to extract features from the video to be analyzed to obtain a multimodal feature sequence.

[0028] In this embodiment, the feature extraction model includes a large visual model, an audio model, and a language model; the multimodal feature sequence includes a visual feature sequence, an audio feature sequence, and a text feature sequence. The large visual model, audio model, and language model can employ existing deep learning models. Preferably, the large visual model can employ a CLIP (Contrastive Language-Image Pre-Training) model or a ViT (Vision Transformer), the audio model can employ PANNs (pre-trained audio neural networks), and the language model can employ BERT (Bidirectional Encoder Representation from Transformers). Existing training methods can be used to train these models, and no limitations are imposed here.

[0029] Specifically, the video to be analyzed is preprocessed to obtain time-aligned video frames, audio segments, and text transcribed from the audio. Specifically, the video to be analyzed is segmented into frames using a preset static or dynamic sampling rate; audio tracks are separated and extracted from the video, and noise reduction and normalization are performed on the audio tracks to obtain audio segments; ASR (Automatic Speech Recognition) technology is used to process the audio extracted from the video to obtain the corresponding text. Furthermore, the video frames, audio segments, and text are aligned on the timeline to ensure the synchronization of subsequent feature fusion. Existing methods can be used for time alignment, and no restrictions are imposed here.

[0030] Furthermore, the video frame is input into the trained visual model to extract the visual feature sequence of the video frame. This visual feature sequence includes visual features such as objects, scenes, and facial emotions, and can be represented as follows: ,in, Represents a sequence of visual features. Indicates the first The visual features of a frame; similarly, by inputting an audio segment into a trained audio model, an audio feature sequence is extracted. This audio feature sequence includes audio features such as speech emotion, voiceprint, and background music genre, which can be represented as... ,in, Represents an audio feature sequence. Indicates the first The audio features of the frame; similarly, the transcribed text is input into the trained language model for word segmentation and encoding to extract a text feature sequence, which includes keywords, topics, and other feature text, and can be represented as... ,in, Represents a text feature sequence. Indicates the first Textual features of the frame.

[0031] By extracting textual, audio, and visual features from the video to be analyzed, the semantic gap problem that occurs in single-modal analysis can be avoided in subsequent analysis, thereby improving the accuracy and reliability of video analysis.

[0032] S3. Input the multimodal feature sequence into the preset video analysis model to obtain the prediction results.

[0033] In this embodiment, the video analysis model includes a temporal encoder, a multi-layer cross-modal encoder, and multiple task prediction heads. The temporal encoder can employ 1D-CNN layers or a single-modal Transformer layer, the cross-modal encoder uses a cross-modal Transformer, and the task prediction heads include a first task prediction head and a second task prediction head.

[0034] In one embodiment, a multimodal feature sequence is input into a preset video analysis model to obtain a prediction result, specifically including the following steps.

[0035] S31. Input the multimodal feature sequences into the corresponding time encoders to obtain the context-encoded multimodal encoded sequences.

[0036] In this embodiment, the multimodal coding sequence includes a visual feature coding sequence, an audio feature coding sequence, and a text feature coding sequence, and the temporal encoder includes a first temporal encoder, a second temporal encoder, and a third temporal encoder.

[0037] Specifically, the visual feature sequence is input into the first temporal encoder to obtain the visual feature encoded sequence; the audio feature sequence is input into the second temporal encoder to obtain the audio feature encoded sequence; and the text feature sequence is input into the third temporal encoder to obtain the text feature encoded sequence. The first, second, and third temporal encoders are independent of each other. The first temporal encoder is used to capture the temporal dependencies within the visual feature sequence, the second temporal encoder is used to capture the temporal dependencies within the audio feature sequence, and the third temporal encoder is used to capture the temporal dependencies within the text feature sequence.

[0038] S32. Input the multimodal coding sequence into the multilayer cross-modal encoder for attention-weighted fusion to obtain the fused feature sequence.

[0039] In one embodiment, the fused feature sequence can be represented as ,in, Represents the fused feature sequence. Indicates the first The frame fusion features are a highly condensed joint temporal sequence that incorporates information from all modalities. Furthermore, the multilayer cross-modal encoder employs a stacked design, combining residual connections and layer normalization.

[0040] Specifically, in each layer of the cross-modal encoder, a query vector is constructed using one modal coding sequence from the multimodal coding sequence, and key and value vectors are constructed using other modal coding sequences. Then, based on the query vector, key vector, and value vector, cross-modal attention is computed using scaled dot product attention.

[0041] In one embodiment, if in a certain layer of a cross-modal encoder, a query vector is constructed using visual feature encoding sequences, and key and value vectors are constructed using audio and text feature encoding sequences, then the visual attention to audio and text (i.e., cross-modal attention) is as follows:

[0042] In the formula, For cross-modal attention; The query vector is derived from a sequence of visual feature codes. The key vector is derived from the concatenation of the audio feature encoding sequence and the text feature encoding sequence; It is a value vector, derived from the concatenation of audio feature encoding sequences and text feature encoding sequences; is the scaling factor, which can be the dimension of the key vector; Softmax() is the Softmax activation function; T is the matrix transpose operation.

[0043] In the above embodiments, the visual feature encoding sequence is used as the query vector, the audio feature encoding sequence and the text feature encoding sequence are concatenated to obtain the key vector, and the audio feature encoding sequence and the text feature encoding sequence are concatenated to obtain the value vector.

[0044] In optional embodiments, existing methods can also be used to construct the query vector, key vector, and value vector, such as performing a linear transformation on the input feature sequence to obtain the query vector, key vector, and value vector. Furthermore, operations following the acquisition of cross-modal attention can be performed using existing methods, and are not limited here.

[0045] Compared to traditional simple weighting or concatenation methods, this approach, which constructs a query vector using one modality coding sequence and key and value vectors using other modality coding sequences to calculate attention, enables video analysis models to learn complex cross-modal associations. For example, when a smiling face appears in the video (corresponding to visuals) and the background music (corresponding to audio) becomes cheerful, the probability of user interaction increases. This approach accurately captures complex and non-linear relationships between modalities, thereby improving the reliability of the generated analysis reports.

[0046] S33. Input the fused feature sequence into multiple parallel task prediction heads to obtain the prediction results.

[0047] In this embodiment, the prediction results include user retention rate and the probability of interactive events. The first task prediction head can use a regression network layer to predict the user retention rate at different time points, i.e., the user retention rate curve; the second task prediction head can use a classification network layer to predict the probability of each interactive event.

[0048] Specifically, the fused feature sequence is input into the first task prediction head to obtain the user retention rate at different time points; in one embodiment, the process of obtaining the user retention rate can be expressed as:

[0049] In the formula, For the first The discrete timestamp corresponding to the frame, i.e., the time point. For at a certain point in time User retention rate Indicates the first Frame fusion features Input to regression network layer middle.

[0050] Simultaneously, the fused feature sequence is input into the second task prediction head to obtain the probability of each interaction event; in one embodiment, the process of obtaining the probability of each interaction event can be expressed as:

[0051] In the formula, For at a certain point in time The probability of each interactive event. It is the Sigmoid activation function. Indicates the first Frame fusion features Input to classification network layer middle.

[0052] It should be noted that MLP, or Multilayer Perceptron, also known as a fully connected neural network layer, is a common type of feedforward neural network. Its internal processing is described in existing technologies and will not be repeated here.

[0053] Because the fused feature sequence input to the task prediction head incorporates information from all modalities, it exhibits greater adaptability and higher prediction accuracy when predicting video content with diverse styles. Furthermore, by using user retention rate and the probability of interactive events as prediction targets, the correlation between the analysis results and actual business metrics is enhanced, ensuring that the generated analysis report has practical guiding value and providing a reliable basis for users' subsequent decision-making or adjustments to video content.

[0054] In one embodiment, the method of the present invention further includes: training and optimizing the video analysis model using the composite loss calculated by the preset multi-objective composite loss function, specifically including the following steps.

[0055] S100, determine retention prediction loss, interactive event prediction loss, and explainable attribution loss.

[0056] In one embodiment, retention prediction loss can be calculated using the Huber loss function; interaction event prediction loss can be calculated using a binary cross-entropy loss function with class weights; and interpretable attribution loss can be calculated using KL divergence.

[0057] In one embodiment, the formula for calculating the retention prediction loss is:

[0058] In the formula, To retain the predicted loss, This refers to the number of time points, or the number of video frames. For at a certain point in time User retention rate This corresponds to the actual user retention rate. Here is the Huber loss function. The preset error threshold can be set manually.

[0059] By employing the Huber loss function to calculate retention prediction loss, the sensitivity of mean squared error to noise and the stability of mean absolute error can be balanced, thereby improving the accuracy and stability of user retention rate prediction.

[0060] In one embodiment, the expression for calculating the interactive event prediction loss is:

[0061] In the formula, Predicting losses for interactive events For the types of interactive events, For the first The category weights of interactive events are either manually set or obtained through training. For the corresponding actual number The probability of a certain type of interactive event. For the predicted first The probability of a certain type of interactive event.

[0062] It should be noted that interactive event prediction is a multi-label classification problem. By employing a binary cross-entropy loss function with class weights to calculate the interactive event prediction loss, the problem of imbalanced positive and negative samples is resolved. At the same time, by assigning class weights to different interactive events, the prediction results are made more consistent with the actual class ratios, avoiding being dominated by the number of samples.

[0063] In one embodiment, the expression for calculating the explainable attribution loss is:

[0064] In the formula, To explain attributable loss, Let KL divergence function be used. Fusion features as input The gradient of the prediction result output by the video analysis model; The corresponding key events annotated by experts are in the first... Importance distribution of frames.

[0065] Understandably, a larger gradient indicates a greater contribution of the feature to the prediction result, i.e., greater importance. In one embodiment, the gradient can be calculated by integrating the gradient, where the baseline feature can be a vector of all zeros. Since gradient integration is a prior art technique, the specific calculation process will not be elaborated here. Similarly, KL divergence is also a prior art technique, and will not be elaborated here either.

[0066] By leveraging integral gradient quantization to quantize the contribution of input features to the decision-making of video analysis models, the interpretability of video analysis models is enhanced, addressing the trust issue inherent in such black-box models. Simultaneously, by introducing interpretable attribution loss, the video analysis model can be guided to learn the attribution logic of human experts, making attention weights more closely aligned with causal explanations and improving the accuracy of predictions.

[0067] It's important to note that before training and optimizing the video analytics model, it's necessary to obtain performance data of the sample videos on various playback platforms. This performance data is used as ground truth to calculate retention prediction loss and interaction event prediction loss. This performance data includes user retention rate curves, second-level interaction data (including likes, comments, shares, etc.), and target user profile tags. Essentially, if a video receives a like at a certain point in time, it means the probability of that like event occurring at that time is 1; if no like occurs, the probability is 0.

[0068] In addition, experts in the field labeled the importance distribution of key events in some sample videos at corresponding time points to obtain labeled data. This labeled data is used as ground truth to calculate the explainable attribution loss. For example, when watching sample videos, domain experts manually assigned contribution weights / importance distributions of different modalities for specific high-interaction or high-stream time points. For instance, in a product promotion video, at the 15th second, the user like rate spiked. Experts believed that the user likes were mainly due to the visual content, followed by the narration, while the background music did not play a significant role. Therefore, the importance distribution of the key event likes at the 15th second was labeled as [visual: 0.7, text: 0.2, audio: 0.1].

[0069] S200. The retention prediction loss, interaction event prediction loss, and explainable attribution loss are weighted and summed to obtain the composite loss.

[0070] Specifically, the formula for calculating the composite loss is:

[0071] In the formula, For compound loss, To retain the predicted loss, To explain attributable loss, Predicting losses for interactive events To retain the weights of the predicted loss, Predicting losses for interactive events As the weight for explainable attribution loss, , and Obtained through hyperparameter search or by expert feedback / input.

[0072] S300. Use this composite loss to train and optimize the video analysis model.

[0073] Specifically, based on the composite loss, the backpropagation algorithm is used to train and optimize the video analysis model. Since the backpropagation algorithm is an existing technology, it will not be described in detail here.

[0074] S4. Generate an analysis report based on the prediction results.

[0075] In one embodiment, the prediction results also include key nodes and key node attributions, wherein key nodes include highlight points and dropout points.

[0076] Specifically, it generates visual charts of user retention rate and the probability of interaction events, and matches corresponding optimization suggestions from a pre-set suggestion database based on the attribution of key nodes; the visual charts, the attribution of key nodes, and the optimization suggestions form an analysis report.

[0077] In one embodiment, after obtaining the analysis report, the method of the present invention further includes: outputting the analysis report for experts to review and verify; further, obtaining expert verification instructions for the analysis report, the expert verification instructions being input / triggered by experts, including verification pass and verification fail; if the expert verification instructions indicate verification pass, then pushing the analysis report as the final analysis report to the target user; if the expert verification instructions indicate verification fail, then obtaining the correction data fed back by the experts, quantifying the correction data, and using the quantified correction data for iterative optimization. , and This data can be used to fine-tune the video analysis model, and / or as labeled data. It should be noted that the above process can be applied during the model training and optimization phase, as well as during the model deployment phase.

[0078] For example, if the corrected data provided by the experts is correct... , and The adjustments made will be applied in the next training session, based on the expert's recommendations. , and If the corrected data provided by experts is the attribution of the key nodes, then the corrected data will be used as the labeled data to fine-tune the video analysis model.

[0079] By introducing the aforementioned expert-in-the-loop closed-loop optimization mechanism, the video analytics model can continuously learn and iterate, adapting to the ever-changing platform environment and video content.

[0080] Figure 2 This is a schematic diagram illustrating the structure of the video analysis report generation system according to this embodiment.

[0081] In a second aspect, the present invention also provides a video analysis report generation system. For example... Figure 2 As shown, the system includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, implement the video analysis report generation method described in the first aspect of the present invention.

[0082] The system also includes other components well known to those skilled in the art, such as communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0083] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0084] In the description of this specification, "multiple" means at least two, such as two, three or more, unless otherwise explicitly specified. Furthermore, the steps described above are for clarity only; in implementation, they can be combined into one step or some steps can be broken down into multiple steps, as long as they include the same logical relationships.

[0085] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for generating video analysis reports, characterized in that, include: Obtain the video to be analyzed; The video to be analyzed is subjected to feature extraction using a preset feature extraction model to obtain a multimodal feature sequence; The multimodal feature sequence is input into a preset video analysis model to obtain the prediction result, specifically: The multimodal feature sequences are input into the corresponding temporal encoders to obtain context-encoded multimodal encoded sequences; the multimodal encoded sequences are input into multi-layer cross-modal encoders for attention-weighted fusion to obtain fused feature sequences; the fused feature sequences are input into multiple parallel task prediction heads to obtain prediction results. An analysis report is generated based on the prediction results.

2. The video analysis report generation method according to claim 1, characterized in that, The multimodal encoded sequence is input into a multilayer cross-modal encoder for attention-weighted fusion, including: In each layer of the cross-modal encoder, a query vector is constructed using one modal coding sequence, and key and value vectors are constructed using other modal coding sequences. Cross-modal attention is computed based on query vector, key vector, and value vector.

3. The video analysis report generation method according to claim 2, characterized in that, The multimodal coding sequence includes a visual feature coding sequence, an audio feature coding sequence, and a text feature coding sequence; if a query vector is constructed using the visual feature coding sequence, and a key vector and a value vector are constructed using the audio feature coding sequence and the text feature coding sequence, then the calculation expression for the cross-modal attention is: In the formula, For cross-modal attention; The query vector is derived from a sequence of visual feature codes. The key vector is derived from the concatenation of the audio feature encoding sequence and the text feature encoding sequence; It is a value vector, derived from the concatenation of audio feature encoding sequences and text feature encoding sequences; This is the scaling factor; Softmax() is the Softmax activation function.

4. The video analysis report generation method according to claim 1, characterized in that, The prediction results include user retention rate and the probability of interaction events; the fused feature sequence is input into multiple parallel task prediction heads to obtain prediction results, including: The fused feature sequence is input into the first task prediction header to obtain the user retention rate at different time points; The fused feature sequence is input into the second task prediction head to obtain the probability of each interaction event.

5. The video analysis report generation method according to claim 4, characterized in that, Also includes: Determine the retention prediction loss, the interaction event prediction loss, and the explainable attribution loss; The weighted sum of retention prediction loss, interaction event prediction loss and explainable attribution loss is used to obtain the composite loss. The video analysis model is trained and optimized using the composite loss.

6. The video analysis report generation method according to claim 5, characterized in that, Determine retention prediction loss, interaction event prediction loss, and explainable attribution loss, including: The Huber loss function is used to calculate the retention prediction loss; The interactive event prediction loss is calculated using a binary cross-entropy loss function with category weights; KL divergence can be used to explain the attribution loss.

7. The video analysis report generation method according to claim 6, characterized in that, The formula for calculating the explainable attribution loss is: In the formula, For explainable attribution loss, KL-Divergence() is the KL divergence function. Fusion features as input The gradient of the prediction result output by the video analysis model. The corresponding key events annotated by experts are in the first... The importance distribution of frames, where Softmax() is the Softmax activation function.

8. The video analysis report generation method according to claim 5, characterized in that, Also includes: Obtain expert verification instructions for the analysis report; If the expert verification instruction fails, the corrected data from the expert feedback is obtained, and the weights of retention prediction loss, interactive event prediction loss, and explainable attribution loss are iteratively optimized using the corrected data.

9. The video analysis report generation method according to claim 1, characterized in that, The feature extraction model includes a large visual model, an audio model, and a language model; the multimodal feature sequence includes a visual feature sequence, an audio feature sequence, and a text feature sequence.

10. A video analysis report generation system, characterized in that, It includes a processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the video analysis report generation method according to any one of claims 1-9.

Images