A video understanding method and system based on creative radar

Abstract

The application relates to the field of artificial intelligence and multimedia processing technology, in particular to a video understanding method and system based on a creative radar, which comprises the following steps: acquiring a video segment input by a user, and extracting a multi-modal feature tensor by using a multi-modal encoder; based on a pre-defined creative dimension set, the multi-modal feature tensor is used to calculate the coverage of each creative dimension, and when the coverage of a specific dimension is lower than a preset threshold, it is determined that there is a creative gap; for the dimension with the creative gap, a guiding suggestion for completing the creative intention is generated in combination with a creator cultural background knowledge base; based on the video segment and creator portrait features, a classifier is used to identify the creative motivation type implied by the video; and according to the identified creative motivation type, evaluation feedback meeting risk constraints is generated. According to the scheme of the application, the creator can be effectively helped to complete content optimization, and a more perfect and emotionally connected video analysis process is realized.

Description

Technical Field

[0001] This invention generally relates to the fields of artificial intelligence and multimedia processing technology. More specifically, this invention relates to a video understanding method and system based on creative radar. Background Technology

[0002] In an era of explosive growth in short videos and AI-generated content, the value extraction of video content has evolved from basic tag recognition (such as object detection and scene classification) to in-depth analysis of the creative core (emotional tension, narrative logic, and cultural metaphors). Current technologies primarily rely on the following solutions: (1) Multimodal feature splicing scheme: Use models such as CLIP and VideoMAE to extract video frame, audio and subtitle features respectively, and generate content summary by simple splicing or weighted fusion. A typical example is Google's Video Intelligence API, which can only recognize predefined categories (such as "beach" and "concert") and cannot parse the level of creative expression.

[0003] (2) Content mining driven by preset rules: Threshold judgment is based on manually defined creative dimensions (such as "pacing" and "color temperature"), which lacks the ability to dynamically perceive implicit creative intentions. For example, Adobe Premiere's AutoReframe function only optimizes the composition of the picture and ignores the narrative depth.

[0004] Therefore, one of the key research directions is how to address the problem that existing video understanding technologies mainly rely on shallow labels (such as object detection) and utilitarian data indicators (such as completion rate), and cannot understand the deep creative intentions and cultural connotations of videos. Summary of the Invention

[0005] To address the technical problem that existing video understanding technologies mainly rely on shallow labels and utilitarian data indicators, and are unable to understand the deep creative intentions and cultural connotations of videos, this invention provides solutions in the following aspects.

[0006] In a first aspect, the present invention provides a video understanding method based on creative radar, comprising: acquiring a video segment input by a user and extracting a multimodal feature tensor using a multimodal encoder; calculating the coverage of each creative dimension based on a predefined set of creative dimensions using the multimodal feature tensor, and determining that a creative gap exists when the coverage of a specific dimension is lower than a preset threshold; generating guiding suggestions for supplementing creative intentions by combining a creator's cultural background knowledge base for the dimensions with creative gaps; identifying the types of creative motivations implicit in the video using a classifier based on the video segment and creator profile features; and generating evaluation feedback that meets risk constraints based on the identified types of creative motivations, wherein the evaluation feedback is used to guide the creator to optimize video content.

[0007] In one embodiment, extracting a multimodal feature tensor using a multimodal encoder includes: extracting visual, audio, and text features separately using a multimodal encoder to generate a fused multimodal feature tensor.

[0008] In one embodiment, the set of creative dimensions includes at least emotional depth, narrative coherence, density of cultural symbols, audiovisual innovation, and metaphor intensity.

[0009] In one embodiment, calculating the coverage of each creative dimension includes: performing a global average pooling operation on the multimodal feature tensor, inputting the obtained pooling result into the classification weight layer corresponding to each creative dimension, and performing Sigmoid activation on the output of the classification weight layer to obtain the coverage.

[0010] In one embodiment, the coverage rate is calculated using the following formula:

[0011] In the formula, For the i-th creative dimension coverage, For dimension weights, For the Sigmoid function, For multimodal feature tensors, For global average pooling, For the i-th creative dimension Dimensional bias.

[0012] In one embodiment, the classifier is a motivational classifier based on a graph neural network, and its input also includes a cultural symbol knowledge base.

[0013] In one embodiment, an evaluation feedback that meets risk constraints is generated based on the identified creative motivation type, including: using the HateBERT model to detect text toxicity scores and calculating the cultural fit corresponding to the creative motivation type based on a cultural symbol knowledge base; and performing threshold determination on the text toxicity scores and cultural fit to determine the evaluation feedback.

[0014] In one embodiment, threshold determination is performed on the text toxicity score and cultural fit to determine the evaluation feedback, including: generating corresponding evaluation feedback in response to a text toxicity score being lower than a preset toxicity threshold and a cultural fit score with the corresponding creative motivation being higher than a preset fit threshold.

[0015] In one embodiment, the preset toxicity threshold is 0.1 and the preset fit threshold is 0.85.

[0016] In a second aspect, the present invention also provides a video understanding system based on creative radar, comprising: a processor; and a memory storing computer program instructions, wherein when the processor executes the computer program instructions, it implements a video understanding method based on creative radar as described in one or more of the foregoing embodiments.

[0017] The beneficial effects of this invention are as follows: According to the solution of this invention, in the early stages of video analysis, gaps in the integrity of creative expression can be automatically identified, thereby generating guiding suggestions to help creators complete their creative intentions, breaking through the limitations of traditional static tagging systems. Simultaneously, based on video clips and creator profile features, a classifier is used to identify the types of creative motivations implicit in the video and generate evaluation feedback that meets risk constraints, effectively helping creators to optimize content and achieve more complete and emotionally connected video analysis and processing. Attached Figure Description

[0018] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein: Figure 1 This is a flowchart illustrating a video understanding method 100 according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the video understanding processing hierarchy according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of a creative radar detection layer according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the structural block diagram of a video understanding system based on creative radar according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This solution proposes a comprehensive approach that integrates creative radar detection with cultural perception feedback. First, through multimodal feature extraction, the "creative radar" mechanism dynamically scans the integrity of videos in dimensions such as emotion, narrative, and metaphor, automatically identifying gaps in creative expression. Then, by combining cultural knowledge graphs and graph neural networks, it deeply analyzes the creator's underlying motivations (such as cultural inheritance and social critique), ultimately generating non-data-driven feedback that can both fill creative gaps and establish emotional resonance with the creator.

[0021] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart illustrating a video understanding method 100 according to an embodiment of the present invention.

[0023] like Figure 1 As shown, in step S101, multimodal feature extraction is performed. Specifically, a video segment input by the user is acquired, and a multimodal feature tensor is extracted using a multimodal encoder. In some embodiments, visual, audio, and textual features can be extracted separately using a multimodal encoder to generate a fused multimodal feature tensor.

[0024] In step S102, a creative dimension gap is detected. Specifically, based on a predefined set of creative dimensions, the coverage rate of each creative dimension is calculated using a multimodal feature tensor. When the coverage rate of a specific dimension is lower than a preset threshold, a creative gap is determined to exist. This set of creative dimensions includes at least emotional depth, narrative coherence, cultural symbol density, audiovisual innovation, and metaphor intensity.

[0025] In some embodiments, when calculating the coverage of each creative dimension, a global average pooling operation can be performed on the multimodal feature tensor, the pooling result can be input into the classification weight layer corresponding to each creative dimension, and the output of the classification weight layer can be activated by Sigmoid to obtain the coverage.

[0026] Specifically, the formula for calculating coverage is:

[0027] In the formula, For the i-th creative dimension coverage, For dimension weights, For the Sigmoid function, For multimodal feature tensors, For global average pooling, For the i-th creative dimension Dimensional bias.

[0028] In step S103, gap guidance information is generated. Specifically, for the dimensions where creative gaps exist, guiding suggestions are generated to complete the creative intent, based on the creator's cultural background knowledge base.

[0029] In step S104, creative motivation is identified. Specifically, based on video clips and creator profile features, a classifier is used to identify the type of creative motivation implicit in the video. This classifier is a motivation classifier based on a graph neural network, and its input also includes a cultural symbol knowledge base.

[0030] In step S105, feedback is generated. Specifically, based on the identified type of creative motivation, evaluation feedback that meets risk constraints is generated. This evaluation feedback is used to guide creators in optimizing video content.

[0031] In some embodiments, when generating evaluation feedback that meets risk constraints, the HateBERT model can be used to detect text toxicity scores, and the cultural fit corresponding to the creative motivation type can be calculated based on a cultural symbol knowledge base. Thresholds are applied to the text toxicity score and cultural fit to determine the evaluation feedback. Specifically, in response to a text toxicity score lower than a preset toxicity threshold and a cultural fit score higher than a preset fit threshold for the corresponding creative motivation, corresponding evaluation feedback is generated. The preset toxicity threshold can be set to 0.1, and the preset fit threshold can be set to 0.85.

[0032] The above describes the solution of the present invention. The solution of the present invention will be described in detail below with reference to specific embodiments.

[0033] Figure 2 This is a schematic diagram illustrating the video understanding processing hierarchy according to an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the principle of a creative radar detection layer according to an embodiment of the present invention.

[0034] like Figure 2 As shown, the above processing is mainly implemented through four levels.

[0035] (1) Input layer. The input layer mainly receives the raw video stream and user interaction behavior, such as the video segment V input by the user.

[0036] (2) Creative Radar Detection Layer. In the early stages of video analysis, the Creative Radar Agent automatically identifies gaps in the completeness of creative expression (such as missing emotional layers or unexplained cultural symbols), generates guiding questions to help creators complete their creative intentions, and breaks through the limitations of the traditional static labeling system.

[0037] like Figure 3As shown, the creative radar detection layer is mainly composed of an engine architecture with at least two parts. The first is a dynamic dimension detector, which scans creative dimensions of 12+ in real time. The second is a gap-guided engine, which calculates the dimension coverage and triggers when the dimension coverage is less than 0.3.

[0038] Specifically, firstly, multimodal feature extraction and spatiotemporal feature tensor generation are performed using a dynamic dimension detector.

[0039] In the initial stage of video analysis, it is necessary to first perform multi-dimensional feature encoding on the input video clips in order to capture their basic information at the visual, auditory, and textual levels.

[0040] Let the input video segment be V. To overcome the limitations of single-modal information, this embodiment employs a multimodal encoder (MM-Encoder) to process different channel data in the video. Specifically, the I3D (Inflated 3DConvNets) network is used to extract visual features Vvis, the VGGish network is used to extract audio features Vaud, and the BERT model is used to extract features Vtxt for subtitles or speech-to-text transcription.

[0041] After extraction, the three features described above are aligned and concatenated to generate a unified spatiotemporal feature tensor T. This process can be represented by the following formula: T=MM-Encoder(V)=[I3D(Vvis);VGGish(Vaud);BERT(Vtxt)] Where d is the feature dimension, and in this embodiment, the preferred dimension d is 768 to balance computational efficiency and feature representation capability. [Symbol] ; The symbol ] represents a tensor concatenation operation. In this way, the system can simultaneously preserve the video's visual dynamics, background music atmosphere, and linguistic semantic information, providing a data foundation for subsequent in-depth creative analysis.

[0042] Next, dynamic detection and gap guidance in the creative dimension.

[0043] After obtaining the spatiotemporal feature tensor T, the system needs to further identify the video's performance in predefined creative dimensions and automatically detect gaps in the integrity of creative expression.

[0044] Predefine a set of creative dimensions D, which includes, but is not limited to, {emotional depth, narrative coherence, density of cultural symbols, audiovisual innovation, metaphorical intensity, rhythmic tension, color symbolism, spatial hierarchy, temporal metaphor, cross-cultural resonance, social critique, and personal healing value}. For each dimension d in set D... i Calculate its coverage c di .

[0045] Coverage is calculated to quantify the intensity of video content's performance in this dimension, and it is based on the following formula:

[0046] In the formula, For the i-th creative dimension The coverage rate ranges from 0 to 1. If the value is lower than a preset threshold (e.g., 0.3), it is determined that there is a creative gap in this dimension, thereby triggering the guide engine. For dimension weights, For dimensions A dedicated weight matrix for training is used to measure the input video features against the current specific creative dimension. The strength of the correlation between them. The Sigmoid function maps the result of linear calculation to the (0,1) interval, making it a probability or proportion value, which is convenient for use as a threshold for coverage and thus intuitively represents the coverage. For multimodal feature tensors, For global average pooling, GAP( The expression ) represents the Global Average Pooling operation, used to compress the spatiotemporal feature tensor T into a global feature vector. It aggregates the input multimodal spatiotemporal feature tensors, compresses the spatiotemporal dimensions, and extracts the global feature vector for computation with the dimension weight vector. For the i-th creative dimension Dimensional bias.

[0047] Based on the calculated coverage The system introduces a "gap-guided engine" mechanism. A gap detection threshold τ is set. gap In this embodiment, τ is preferred. gap =0.3. When the coverage of a certain dimension is lower than this threshold (i.e., less than 0.3), it indicates that the video's expression in that creative dimension is significantly lacking or has not been fully explored. At this time, the system triggers the Large Language Model (LLM) to generate guiding questions:

[0048] in, The system includes a pre-built knowledge base of creators' cultural backgrounds, covering cultural norms from over 200 countries and ethnic groups. Understandably, by introducing a threshold of 0.3, the system can filter out dimensions with acceptable expression, focusing on obvious creative weaknesses. For example, if the coverage rate for the "cultural symbol density" dimension is detected as 0.18, the system, combined with the knowledge base, might generate the following prompt: "Your video is weak in the [cultural symbol density] dimension. The shot of the withered tree could strengthen the metaphor: should we add close-ups of raindrops to symbolize hope and rebirth? (Refer to the classic technique in *The Shawshank Redemption*)."

[0049] (3) Value feedback layer.

[0050] Building upon an understanding of the video's creative concept, the system needs to further identify the creator's deeper motivations and generate feedback with a more profound level of understanding.

[0051] First, motivation identification. To understand "why the creator made this video," an input vector X containing multi-source features is constructed, including video content features, geolocation (GeoTag), posting time context, and creator profile. These features are then fed into a graph neural network (GNN) that incorporates a pre-trained cultural symbol knowledge graph (containing 58,217 nodes).

[0052] X=[ T GeoTag( V TimeContext( V );CreatorProfile] The input includes not only spatiotemporal features T, but also geotags GeoTag, time context TimeContext, and creator profile CreatorProfile.

[0053] Secondly, probability calculation. A culturally enhanced graph neural network (GNN) is used to classify creative motivations. In this embodiment, the GNN model incorporates a pre-trained cultural symbol knowledge graph, Gcul (containing tens of thousands of nodes and millions of edges). The calculation process for the output motivation probability P(y|X) is as follows: P (y∣X)=GNN(G cul ⊕X) The purpose of calculating motivation probability is to determine the probability that a video belongs to a specific creative motivation category y given multivariate input features X. By calculating the probability distribution, the most likely deep motivation of the video is determined (e.g., P(cultural heritage|X) = 0.85), thereby generating feedback that matches that motivation. culFor a pre-trained cultural symbol knowledge graph, the output category y∈Y={cultural heritage, social critique, personal healing, historical record, aesthetic experiment} is defined. This cultural symbol knowledge graph is an external knowledge base that stores cultural symbols and their relationships from around the world; for example, it contains 58,217 nodes and 1,248,935 edges. By introducing G... cul The model can understand the symbolic meaning within a specific cultural context. For example, it can identify that a particular garment represents "cultural heritage" rather than simply "clothing display," and that the motivation behind a video is "cultural heritage" (highest probability) rather than simply "life recording." ⊕ represents the fusion operation between the graph features and the input vector, used to inject or map specific video features X onto the grand cultural knowledge graph G. cul In this context, the technology can be implemented through feature alignment or node attribute enhancement operations in graph neural networks, allowing video features to propagate and be inferred within the structure of the cultural graph.

[0054] Finally, feedback generation and constraints are implemented based on the identified motivation y. The system utilizes a large language model to generate feedback R and imposes strict generative feedback constraints. In this embodiment, an adversarial discriminator mechanism is introduced when generating feedback text, imposing three constraints: toxicity constraint (Toxicity < 0.1): ensuring no offensive language; cultural fit (CulturalFit > 0.85): ensuring the feedback conforms to the identified "cultural heritage" context; and de-utilitarian constraint (DataDependency < 0.2): prohibiting the generation of data-driven suggestions such as "suggesting fast cuts to improve completion rate".

[0055] Based on this, the generated feedback text is: R=LLM value (y V, G cul ) Where R represents the final generated feedback text, LLM value ( ) is a large language model focused on value feedback, serving as an engine for generating feedback text.

[0056] The generation process must simultaneously satisfy the following constraints:

[0057] Among them, Toxicity is a toxicity detection metric, detected using the HateBERT model. A value less than 0.1 indicates that the generated text is safe and harmless, preventing the generation of text containing hate speech, insults, or inciting harmful content. CulturalFit is the cultural fit between the feedback content and the identified creative motivation type. It can be detected using a cultural fit model. A value greater than 0.85 indicates that the generated text highly matches the identified creative motivation. Ensure the accuracy of the feedback. If the video motivation is a serious "historical record," the feedback cannot be lighthearted or humorous; if the motivation is "personal healing," the feedback needs to reflect empathy. DataDependency is a data dependency metric used to limit utilitarian expressions based on traffic data. This part can be detected using a data dependency detection model. This metric can filter out mechanical indicators (such as suggestions to speed up editing to improve completion rate, high dropout rate in the first 3 seconds), preserving descriptions about emotion, narrative, and cultural values.

[0058] The aforementioned constraints ensure that the generated feedback is neither toxic nor highly relevant to the cultural context, and does not rely on tedious data listings (such as "completion rate"), thus providing empathetic feedback such as "silently conveying the pain of cultural discontinuity." For example, by filtering utilitarian expressions through an adversarial discriminator, the generated example is: "The trembling fingers of the old man touching the family genealogy in the shot (00:47) silently convey the pain of cultural discontinuity. This restrained narrative has a stronger power of inheritance than straightforward explanation—it allows young viewers to touch the warmth of history themselves."

[0059] (4) Output layer. The enhanced suggestions are output through the above scheme.

[0060] Through the above embodiments, the present invention not only points out what the video "lacks" (creative gaps), but also deeply understands what the video "wants to express" (cultural motivations), thus providing comprehensive guidance from technical repairs to spiritual resonance.

[0061] Figure 4 This is a schematic diagram illustrating the structural block diagram of a video understanding system based on creative radar according to an embodiment of the present invention.

[0062] This invention also provides a video understanding system. For example... Figure 4 As shown, the system includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, implement a video understanding method based on creative radar as described above.

[0063] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and therefore will not be described in detail here.

[0064] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0065] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A video understanding method based on creative radar, characterized in that, include: The system acquires video clips input by the user and extracts multimodal feature tensors using a multimodal encoder. Based on a predefined set of creative dimensions, the coverage of each creative dimension is calculated using the multimodal feature tensor. When the coverage of a specific dimension is lower than a preset threshold, a creative gap is determined to exist. The set of creative dimensions includes at least emotional depth, narrative coherence, cultural symbol density, audiovisual innovation, and metaphor intensity. Calculate the coverage of each creative dimension, including: A global average pooling operation is performed on the multimodal feature tensor, and the resulting pooling result is input into the classification weight layer corresponding to each creative dimension. The output of the classification weight layer is activated by Sigmoid to obtain the coverage rate. For dimensions where there are creative gaps, guiding suggestions are generated by combining the creator's cultural background knowledge base to complete the creative intent; Based on the video clips and creator profile features, a classifier is used to identify the types of creative motivations implied in the videos; Based on the identified types of creative motivations, evaluation feedback that meets risk constraints is generated, and the evaluation feedback is used to guide creators to optimize video content; The HateBERT model was used to detect the toxicity score of texts, and the cultural fit of creative motivation types was calculated based on a cultural symbol knowledge base. In response to a text toxicity score below a preset toxicity threshold and a cultural fit score above a preset fit threshold for the corresponding creative motivation, corresponding evaluation feedback is generated.

2. The video understanding method based on creative radar according to claim 1, characterized in that, Extracting multimodal feature tensors using a multimodal encoder includes: Visual, audio, and text features are extracted separately using a multimodal encoder to generate a fused multimodal feature tensor.

3. The video understanding method based on creative radar according to claim 1, characterized in that, The formula for calculating coverage is: In the formula, For the i-th creative dimension coverage, For dimension weights, For the Sigmoid function, For multimodal feature tensors, For global average pooling, For the i-th creative dimension Dimensional bias.

4. The video understanding method based on creative radar according to claim 1, characterized in that, The classifier is a motivational classifier based on graph neural networks, and its input also includes a cultural symbol knowledge base.

5. A video understanding method based on creative radar according to claim 1, characterized in that, The preset toxicity threshold is 0.1, and the preset fit threshold is 0.

85.

6. A video understanding system based on creative radar, characterized in that, include: processor; A memory storing computer program instructions, which, when executed by the processor, implement a video understanding method based on creative radar as described in any one of claims 1-5.

Images