Highlight bright spot extraction method and system based on multi-modal model

Abstract

The invention relates to the technical field of image data extraction, in particular to a highlight bright spot extraction method and system based on a multi-modal model, and the method comprises the steps: carrying out the fragment segmentation of a video material after responding to the input of the video material, and obtaining a plurality of overlapped sub-fragments; inputting the plurality of overlapped sub-fragments into a preset multi-modal large model for joint feature extraction to obtain mixed features, extracting historical fragment features and current fragment features from the mixed features, and inputting the historical fragment features and the current fragment features into a preset time-space cross attention mechanism model to obtain a time-space cross attention mechanism model; an attention score matrix in the time dimension is obtained through calculation, and finally, highlight fragments corresponding to elements with scores larger than a preset threshold value in the attention score matrix serve as highlight bright spots. Compared with the prior art, the method provided by the invention solves the technical problems of poor pertinence and low extraction efficiency in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of image data extraction, and more particularly to a method and system for extracting highlights based on a multimodal model. Background Art

[0002] With the explosive growth of short video platforms and social media, advertising content often needs to capture viewers' attention in a very short time. This ability often relies on a few visually impactful or sensory-empathetic images. Traditionally, advertising image extraction requires professionals to extract highlights from video footage to serve as highlight images or clips for advertising and promotion.

[0003] The traditional highlight extraction method is time-consuming and labor-intensive. To improve efficiency, a Chinese patent application, publication number CN119763014A, discloses a deep learning-based video content analysis method, device, medium, and product. This method primarily relies on a spatiotemporal pyramid attention network and a dynamic contrastive learning framework to extract keyframe images. However, this method, when directly applied to extract highlight clips in advertising, has the following drawbacks:

[0004] 1. Lack of targeting, making it difficult to identify the highlights of advertising video materials.

[0005] 2. The inference engine uses a black-box model, which makes it impossible to optimize the performance of the spatiotemporal pyramid attention network. When migrating to advertising video analysis, it is difficult to set the inference rule engine, resulting in an increase in the number of training rounds and a decrease in the overall efficiency of highlight extraction.

[0006] Therefore, the existing technology has the problems of poor targeting and low extraction efficiency in the application of the advertising field. Summary of the Invention

[0007] In order to solve the technical problems of poor pertinence and low extraction efficiency in the above-mentioned prior art, the present invention discloses a method and system for extracting highlight points based on a multimodal model.

[0008] In a first aspect, the present invention discloses a method for extracting highlights based on a multimodal model, comprising:

[0009] In response to an input of a video material, the video material is segmented to obtain a plurality of overlapping sub-segments;

[0010] Input multiple overlapping sub-segments into a preset multimodal large model for joint feature extraction to obtain hybrid features;

[0011] Extracting historical segment features and current segment features from the mixed features;

[0012] Input the historical segment features and the current segment features into the preset spatiotemporal cross attention mechanism model to calculate the attention score matrix in the time dimension;

[0013] The highlight fragments or highlight nodes corresponding to the elements in the attention score matrix whose scores are greater than the preset threshold are taken as highlight points.

[0014] Beneficial Effects: Targeting the extraction of highlights from video material in the advertising field, the method of the present invention utilizes a large multimodal model to deeply understand the video material, enabling the complementarity of information from different modalities, and enhancing the expressive power of features and the pertinence of feature extraction. Furthermore, the method of the present invention utilizes a spatiotemporal cross-attention mechanism model to calculate the feature attention score matrix, using the attention score matrix as a basis to more efficiently screen highlights, achieving precise focusing and positioning in the temporal dimension, and associating them with the context. Compared to existing technologies, the method of the present invention addresses the technical issues of poor pertinence and low extraction efficiency in existing technologies.

[0015] Preferably, multiple overlapping sub-segments are input into a preset multimodal large model for joint feature extraction, and the specific expression of the hybrid feature is obtained as follows:

[0016]

[0017] Where H i represents the i-th mixed feature of the video material, VideoLLM represents the multimodal large model, S i represents the i-th overlapping sub-segment of the video material, represents the cross-modal attention fusion operator, f vision (V i ) represents the i-th visual modality feature in the video material, f audio (A i ) represents the i-th audio modal feature in the video material, f text (T i ) represents the i-th text or subtitle modality feature in the video material.

[0018] Beneficial effects: For the extraction of mixed features, the method of the present invention extracts features from three major modalities: visual modality, audio modality and subtitle modality. Compared with traditional single-modal feature extraction (mainly referring to visual modality), the method of the present invention fully considers the particularity of advertising video materials (that is, on the basis of video performance, exaggerated or inductive voice emphasis or artistic text will be used), and can extract effective features more accurately and specifically.

[0019] Preferably, before inputting the multiple overlapping sub-segments into a preset multimodal large model for joint feature extraction to obtain mixed features, the method of the present invention further includes:

[0020] Define the fragment overlap ratio of multiple overlapping sub-fragments;

[0021] According to the overlap ratio, long-range dependency constraints are set for adjacent overlapping sub-segments.

[0022] Preferably, the expression of the long-range dependency constraint is:

[0023] S i+1 ∩S i =α·|S i |

[0024] In the formula, α represents the fragment overlap ratio, S i+1 represents the i+1th overlapping sub-segment of the video material, S i+1 For S i Adjacent overlapping sub-slices that are adjacent in time.

[0025] Beneficial Effect: By designing the long-range dependency constraint, the method of the present invention can carry a certain amount of previous context information when processing each sub-segment. This allows for better capture of long-range dependencies in the data during subsequent analysis and modeling operations.

[0026] Preferably, before inputting the multiple overlapping sub-segments into a preset multimodal large model for joint feature extraction to obtain mixed features, the method of the present invention further includes:

[0027] Configure the Highlight Inference Engine, which is used to drive the multimodal large model to focus on the plot continuity of video materials.

[0028] Furthermore, the Gaoguang inference engine includes a Prompt dynamic evolution mechanism, which is specifically expressed as follows:

[0029]

[0030] Where, P k Indicates the Prompt generated by the kth iteration, P base Represents the basic Prompt, Concat represents the concatenation function, represents the summary features of the previous reasoning, Used to summarize the key information of multimodal data in the previous reasoning process.

[0031] Beneficial effect: In order to improve the data processing capability of the multimodal large model, the method of the present invention configures a high-light inference engine for the multimodal large model, which can effectively guide the model to focus on key information and improve the performance of the model.

[0032] Preferably, the high-light reasoning engine is further provided with a multimodal reward function for defining a dual-objective optimization, wherein the defined dual-objective optimization includes a coherence reward for adjacent overlapping sub-segments and an audience attraction reward.

[0033] Beneficial effects: The method of the present invention sets up two target optimizations, namely, the consistency reward and the audience attraction reward. Through these two rewards, the content quality of highlights can be enhanced and the user's viewing experience can be improved.

[0034] Furthermore, the definition of the multimodal reward function is:

[0035]

[0036] Where, Indicates the maximization operation of the parameter θ, represents the expectation operator, represents the sum from time step t = 1 to T, γ t represents the discount factor, R cohere (S t ) represents the coherence reward of overlapping sub-segments at time step t, λ represents the hyperparameter, R impact (s t ) represents the audience attractiveness reward of the overlapping sub-segment at time step t.

[0037] Preferably, the spatiotemporal cross attention mechanism model adopts any one of the decomposition-based attention framework, the supervised learning framework or the unified attention framework.

[0038] In the second aspect, the present invention also discloses a highlight bright spot extraction system based on a multimodal model, including a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, the highlight bright spot extraction method based on the multimodal model recorded in the first aspect is implemented.

[0039] The beneficial effects of the present invention are:

[0040] 1. Targeting the extraction of highlights from video material in the advertising field, the method of the present invention utilizes a large multimodal model to deeply understand the video material, enabling the complementarity of information from different modalities, enhancing the expressive power of features and the targeted nature of feature extraction. Furthermore, the method of the present invention utilizes a spatiotemporal cross-attention mechanism model to calculate the feature attention score matrix. This attention score matrix is ​​then used to more efficiently screen highlights, achieving precise focusing and positioning in the temporal dimension, and associating them with the context. Compared to existing technologies, the method of the present invention addresses the technical issues of poor targeting and low extraction efficiency in existing technologies.

[0041] 2. Compared with the existing technology, the method of the present invention configures a high-light inference engine for the multimodal large model, which can effectively guide the model to focus on key information and improve the performance of the model.

[0042] 3. Compared with the existing technology, the method of the present invention sets up two target optimization directions, namely consistency reward and audience attraction reward. Through these two rewards, the content quality of highlights can be enhanced and the user viewing experience can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an illustrative and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0044] Figure 1 Flowchart of a method for extracting highlights based on a multimodal model in Embodiment 1 of the present invention;

[0045] Figure 2 It is a structural diagram of a highlight extraction system based on a multimodal model in the third embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0047] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] Example 1

[0049] like Figure 1 As shown, this embodiment discloses a highlight point extraction method based on a multimodal model, including:

[0050] S10: In response to input of a video material, the video material is segmented to obtain a plurality of overlapping sub-segments.

[0051] In this embodiment, the video material primarily refers to video material in the advertising field. It can also include popular short dramas, product promotion videos, public service announcements, variety shows, or celebrity videos. Overlapping sub-segments refer to sub-segments in the original video sequence data where some content overlaps between the selected sub-segments (i.e., multiple video segments with overlapping content).

[0052] S20: Input the multiple overlapping sub-segments into a preset multimodal large model to perform joint feature extraction to obtain mixed features.

[0053] In this embodiment, the multimodal large model extracts mixed features including visual modality features, audio modality features, and text or subtitle modality features.

[0054] S30: extracting historical segment features and current segment features from the mixed features.

[0055] S40: Input the historical segment features and the current segment features into the preset spatiotemporal cross attention mechanism model to calculate the attention score matrix in the time dimension.

[0056] In this embodiment, the spatiotemporal cross-attention mechanism model adopts a spatiotemporal cross-attention mechanism model based on a unified attention framework. In other embodiments, a spatiotemporal cross-attention mechanism model based on a decomposition attention framework or a supervised learning framework may also be adopted.

[0057] S50: taking the highlight segments or highlight nodes corresponding to the elements in the attention score matrix whose scores are greater than a preset threshold as highlight points.

[0058] It should be explained that video keyframes refer to image data that focuses on the structural and representative nature of the video content, while highlights are a term used in the advertising field, and they mainly refer to climax images or clips that focus on the audience's viewing experience. The attention score matrix is ​​a spatiotemporal cross-attention weight matrix. The score of each element in this weight matrix represents its weight coefficient as a highlight. The higher the score, the higher the possibility that the video clip or node corresponding to the element can be used as a highlight. The highlights obtained by the above method can be a static image of a moment (highlight node) or a short dynamic image (highlight clip). The multiple highlights obtained by the above method are provided as candidate sets to professionals or professional software for editing and integration.

[0059] Through the above steps S10-S50, the method of the present invention uses a multimodal large model to deeply understand the video material, achieving information complementarity among visual, audio, and text modalities, thereby enhancing the expressive power of features and the targeted nature of feature extraction. In addition, the method of the present invention also uses a spatiotemporal cross-attention mechanism model to calculate the attention score matrix of the features, and uses the attention score matrix as a basis to more efficiently screen highlights, thereby achieving precise focusing and positioning of highlights in the time dimension and associating them with the context.

[0060] Through the above technical solution, the method of the present invention solves the technical problems of poor pertinence and low extraction efficiency in the prior art.

[0061] Example 2

[0062] Based on the first embodiment, this embodiment further explains the specific algorithm used in the method of the present invention with respect to the disclosure of the first embodiment.

[0063] In this embodiment, multiple overlapping sub-segments are input into a preset multimodal large model for joint feature extraction, and the specific expression of the hybrid feature is obtained as follows:

[0064]

[0065] Where H i represents the i-th mixed feature of the video material, VideoLLM represents the multimodal large model, S i represents the i-th overlapping sub-segment of the video material, represents the cross-modal attention fusion operator, f vision (V i ) represents the i-th visual modality feature in the video material, f audio (A i ) represents the i-th audio modal feature in the video material, f text (T i ) represents the i-th text or subtitle modality feature in the video material.

[0066] Preferably, Represents the real number space of d dimensions.

[0067] It's important to explain that in advertising, video footage often includes climaxes, inducing moments, or exaggerated emphasis. These elements can manifest themselves in various forms. For example, to provide visual stimulation, unexpected visual fragments may appear during character movements or scene changes; subtitles or commentary may be added to highlight product performance; and audio may be adjusted in volume or through dubbing to enhance empathy. All of these elements can potentially become highlights, and the modal characteristics of these highlights are often multimodal.

[0068] Through the above algorithm design, the method of the present invention adopts VideoLLM as a multimodal large model. This multimodal large model can realize multimodal feature extraction of video materials, and perform attention fusion operations on the above multimodal features to obtain more representative mixed features, thereby realizing multimodal slice coding.

[0069] Preferably, before inputting the multiple overlapping sub-segments into a preset multimodal large model for joint feature extraction to obtain mixed features, the method of the present invention further includes:

[0070] Define the fragment overlap ratio of multiple overlapping sub-fragments;

[0071] According to the overlap ratio, long-range dependency constraints are set for adjacent overlapping sub-segments.

[0072] Specifically, the expression of the above long-range dependency constraint is:

[0073] S i+1 ∩S i =α·|S i |

[0074] In the formula, α represents the fragment overlap ratio, S i+1 represents the i+1th overlapping sub-segment of the video material, S i+1 For S i Adjacent overlapping sub-slices that are adjacent in time.

[0075] In this embodiment, the slice overlap ratio is 0.2.

[0076] The long-range dependency constraints described above achieve the effect of setting a context sliding window. This approach allows a certain amount of previous context information to be included when processing each sub-segment. This allows the model or algorithm to better capture long-range dependencies in the data during subsequent analysis and modeling operations.

[0077] Furthermore, before inputting the multiple overlapping sub-segments into the preset multimodal large model for joint feature extraction to obtain hybrid features, the method of the present invention further includes:

[0078] Configure a high-light inference engine to drive a large multimodal model to focus on the continuity of the plot of video materials.

[0079] Specifically, the above-mentioned Gaoguang reasoning engine includes the Prompt dynamic evolution mechanism and multimodal reward function.

[0080] Among them, the specific expression of Prompt dynamic evolution mechanism is:

[0081]

[0082] Where, P k Indicates the Prompt generated by the kth iteration, P base Represents the basic Prompt, Concat represents the concatenation function, represents the summary features of the previous reasoning, Used to summarize the key information of multimodal data in the previous reasoning process.

[0083] It should be explained that Prompt in this embodiment refers to the input text provided to the artificial intelligence model, which is mainly used to guide the model to generate specific outputs.

[0084] In terms of driving large multimodal models, the design of the Prompt dynamic evolution mechanism has the following advantages:

[0085] First, it can precisely guide model derivation. While large multimodal models are capable of processing data from multiple modalities, such as text, images, and audio, complex tasks require precise guidance to leverage the deeper capabilities of these different modalities. The Prompt dynamic evolution mechanism dynamically adjusts the structure and content of input prompts based on task requirements.

[0086] Second, it can enhance the adaptability of large multimodal models to diverse tasks. Different tasks have significantly different requirements for multimodal data, and the Prompt dynamic evolution mechanism analyzes task characteristics to adaptively generate more appropriate prompts.

[0087] Third, it can improve the efficiency of user interaction with large, multimodal models. The Prompt dynamic evolution mechanism understands user intent in real time and dynamically adjusts prompts, reducing the number of times users have to repeatedly modify their input. In the field of intelligent design, after a designer proposes an initial idea, the mechanism can dynamically optimize the prompt based on the text description or sketch they input, guiding the model to quickly extract multiple highlights that meet the requirements, accelerating the design iteration process and improving interaction efficiency.

[0088] Fourth, it can optimize the fusion effect of large multimodal models. The core of large multimodal models lies in the fusion of data from different modalities, and the Prompt dynamic evolution mechanism can promote the effective integration of information from various modalities by optimizing prompts. For example, in the process of understanding overlapping sub-segments, the mechanism can dynamically generate prompts based on the visual content, audio features, and subtitle text of the overlapping sub-segments, guiding the model to explore the correlation information between the various modalities, such as judging the character's emotions through voice intonation, and understanding the plot development in combination with the picture scene, achieving deeper multimodal fusion and improving the model's ability to understand complex scenes.

[0089] Among them, the specific definition of the multimodal reward function is:

[0090]

[0091] Where, Indicates the maximization operation of the parameter θ, represents the expectation operator, represents the sum from time step t = 1 to T, γ t represents the discount factor, R cohere (s t ) represents the coherence reward of overlapping sub-segments at time step t, λ represents the hyperparameter, R impact (s t) represents the audience attractiveness reward of the overlapping sub-segment at time step t.

[0092] In this embodiment, R cohere It is calculated by the temporal convolutional network TGCN, and R impact Make predictions based on audience attention heatmaps.

[0093] Through the design of the above multimodal reward function, the method of the present invention has the following advantages:

[0094] First, it further improves the performance of large multimodal models. The consistency reward in the reward function ensures that the content generated by the large multimodal model remains logically and semantically coherent at time step t. The audience appeal reward encourages the large multimodal model to extract more compelling highlights, thereby meeting users' demand for high-quality, engaging content.

[0095] Second, it further improves the multimodal fusion capabilities of the multimodal large model. Specifically, the reward function guides the multimodal large model to better integrate this information by comprehensively considering rewards such as coherence and attractiveness.

[0096] Compared with the existing technology, the method of the present invention specifically configures a highlight inference engine based on iterative reinforcement learning for large multimodal models. The highlight inference engine can effectively guide the model to focus on key information, improve the performance of the model, and further improve the targetedness of the method of the present invention and the efficiency of highlight extraction.

[0097] For example, the spatiotemporal cross attention mechanism model expression of this embodiment is as follows:

[0098]

[0099] Where, represents the attention score matrix; Represents the characteristics of historical fragments; Indicates the current segment features; Q t represents the query vector, which is generated based on the information of the current time step t and is used to query between historical and current segment features; It means that the historical segment features and the current segment features are spliced ​​together to form a more comprehensive feature representation; d represents the latitude of the feature.

[0100] Through the design of the above-mentioned spatiotemporal cross attention mechanism model, the method of the present invention has the following advantages:

[0101] First, it achieves temporal alignment across segments. By calculating attention weights, this mechanism can identify relevant features in historical and current segments, thereby achieving temporal alignment between different segments. This is crucial for understanding temporal dependencies and event development in multimodal data.

[0102] Second, it can capture multimodal connections. The spatiotemporal cross-attention mechanism simultaneously considers information in both temporal and spatial dimensions, effectively capturing the complex connections between multimodal data. For example, in video analysis, it can combine visual features, audio features, and text descriptions of video frames to more accurately locate and understand events in the video.

[0103] Third, it can improve the accuracy of highlight location. By focusing on key features and information, this mechanism helps improve the accuracy of multimodal temporal localization. The spatiotemporal cross-attention mechanism can filter out irrelevant information, reduce noise interference, and enable large multimodal models to focus more on features that are useful for localization tasks.

[0104] Example 3

[0105] like Figure 2 As shown, based on Example 1 or 2, this embodiment discloses a highlight bright spot extraction system based on a multimodal model, including a processor and a memory, the memory storing computer program instructions, and when the computer program instructions are executed by the processor, the highlight bright spot extraction method based on the multimodal model recorded in Example 1 or 2 is implemented.

[0106] The system also includes other components familiar to those skilled in the art, such as a communication bus and a communication interface. The configuration and functions of these components are well known in the art and are therefore not described in detail here. In the description of this specification, "plurality" means at least two, such as two, three, or more, unless otherwise specifically defined.

[0107] While several embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous modifications, variations, and alternatives will occur to those skilled in the art without departing from the concept and spirit of the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention.

Claims

1. A highlight extraction method based on a multimodal model, characterized in that: include: In response to input of a video material, the video material is segmented to obtain a plurality of overlapping sub-segments; Input multiple overlapping sub-segments into a preset multimodal large model for joint feature extraction to obtain hybrid features; Extracting historical segment features and current segment features from the mixed features; Input the historical segment features and the current segment features into a preset spatiotemporal cross attention mechanism model to calculate an attention score matrix in the time dimension; The highlight segments or highlight nodes corresponding to the elements in the attention score matrix whose scores are greater than a preset threshold are taken as highlight points.

2. The method for extracting highlights based on a multimodal model according to claim 1, characterized in that: Multiple overlapping sub-segments are input into the preset multimodal large model for joint feature extraction, and the specific expression of the mixed feature is obtained as follows: Where H i represents the i-th mixed feature of the video material, VideoLLM represents the multimodal large model, S i represents the i-th overlapping sub-segment of the video material, represents the cross-modal attention fusion operator, f vision (V i ) represents the i-th visual modality feature in the video material, f audio (A i ) represents the i-th audio modal feature in the video material, f text (T i ) represents the i-th text or subtitle modality feature in the video material.

3. The method for extracting highlights based on a multimodal model according to claim 1, wherein: Before inputting the multiple overlapping sub-segments into a preset multimodal large model for joint feature extraction to obtain mixed features, the method further includes: Define the fragment overlap ratio of multiple overlapping sub-fragments; According to the overlap ratio, long-range dependency constraints of adjacent overlapping sub-segments are set.

4. The method for extracting highlights based on a multimodal model according to claim 3, wherein: The expression of the long-range dependency constraint is: S i+1 ∩S i =α·|S i | In the formula, α represents the fragment overlap ratio, S i+1 represents the i+1th overlapping sub-segment of the video material, S i+1 For S i Adjacent overlapping sub-slices that are adjacent in time.

5. The method for extracting highlights based on a multimodal model according to claim 1, wherein: Before inputting the multiple overlapping sub-segments into a preset multimodal large model for joint feature extraction to obtain mixed features, the method further includes: A highlight inference engine is configured, and the highlight inference engine is used to drive the multimodal large model to focus on the plot continuity of the video material.

6. The method for extracting highlights based on a multimodal model according to claim 5, characterized in that: The high-light inference engine includes a Prompt dynamic evolution mechanism, which is specifically expressed as follows: Where, P k Indicates the Prompt generated by the kth iteration, P base Represents the basic Prompt, Concat represents the concatenation function, represents the summary features of the previous reasoning, Used to summarize the key information of multimodal data in the previous reasoning process.

7. The method for extracting highlights based on a multimodal model according to claim 1, characterized in that: The high-light reasoning engine is also provided with a multimodal reward function for defining a dual-objective optimization, wherein the defined dual-objective optimization includes a coherence reward for adjacent overlapping sub-segments and an audience attractiveness reward.

8. The method for extracting highlights based on a multimodal model according to claim 1, wherein: The definition of the multimodal reward function is: Where, Indicates the maximization operation of the parameter θ, represents the expectation operator, represents the sum from time step t = 1 to T, γ t represents the discount factor, R cohere (S t ) represents the coherence reward of overlapping sub-segments at time step t, λ represents the hyperparameter, R impact (s t ) represents the audience attractiveness reward of the overlapping sub-segment at time step t.

9. The method for extracting highlights based on a multimodal model according to claim 1, wherein: The spatiotemporal cross-attention mechanism model adopts any one of the decomposition-based attention framework, supervised learning framework or unified attention framework.

10. A highlight extraction system based on a multimodal model, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method for extracting highlight points based on a multimodal model according to any one of claims 1 to 9 is implemented.

Images