High-light fragment detection model training method and video processing method
By training a highlight fragment detection model and using a large language model to automatically detect highlight fragments in videos, the problems of unstable quality and low efficiency caused by manual operation are solved, and efficient and quality-controllable video content distribution is achieved.
Patent Information
- Application Number
- CN202611005636.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-07
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, the production of highlight-edited videos relies on manual operation, resulting in unstable quality, long production cycles, and low efficiency, which cannot meet the needs of large-scale video content distribution.
By acquiring video samples labeled with highlight reference information, a large language model is used to extract media materials of various media types, perform encoding processing and multimodal feature fusion, train a highlight fragment detection model, and realize the automated detection and extraction of highlight fragments.
It reduces human intervention, shortens the production cycle, improves video editing efficiency and quality control, and meets the needs of large-scale video content distribution.
Smart Images

Figure CN123123607A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of video processing technology, and in particular to a training method for a highlight fragment detection model and a video processing method. Background Technology
[0002] Extracting highlights from lengthy original videos and automatically generating compact, engaging montage videos has become a key requirement for improving video content distribution.
[0003] Currently, the production process for highlight mashup videos in the industry generally requires manual completion of a series of operations, including highlight clip selection, content understanding, post-production editing, and compositing. This model, which relies on subjective aesthetic judgment and personal experience, results in inconsistent quality of mashup videos, long production cycles, and low efficiency, making it unable to meet the needs of large-scale video content distribution. Summary of the Invention
[0004] To address the aforementioned technical issues, this disclosure provides a training method for a highlight fragment detection model and a video processing method.
[0005] According to one aspect of this disclosure, a method for training a highlight fragment detection model is provided, the method comprising: Acquire video samples; wherein the video samples are marked with specular reference information; wherein the specular reference information is used to represent information of specular segments serving as a reference standard; Based on the first prompt text, a large language model is used to extract first media materials of multiple media types from the video samples. The first media materials are encoded to obtain first multimodal features. Highlight analysis is performed on the video samples based on the first multimodal features to obtain highlight prediction information. Based on the difference between the specular reference information and the specular prediction information, the large language model is iteratively trained; wherein, after completing the iterative training, the large language model is used as a specular fragment detection model.
[0006] According to another aspect of this disclosure, a video processing method is also provided, the method comprising: Obtain the original video; Based on a second prompt text, a highlight fragment detection model extracts second media materials of various media types from the original video. These second media materials are then encoded to obtain second multimodal features. The original video is then analyzed based on these second multimodal features to obtain target highlight information. The highlight fragment detection model is trained using the aforementioned training method. The second prompt text is a task description text used to guide the highlight fragment detection model in performing highlight fragment detection based on the second media materials. Based on the target highlight information, an editing operation is performed on the original video to generate a target video corresponding to the original video.
[0007] According to another aspect of this disclosure, a training apparatus for a highlight fragment detection model is also provided, the apparatus comprising: A sample acquisition module is used to acquire video samples; wherein the video samples are marked with highlight reference information; wherein the highlight reference information is used to represent information of highlight segments as a reference standard; The highlight prediction module is used to extract first media materials of multiple media types from the video sample based on the first prompt text using a large language model, encode the first media materials to obtain first multimodal features, and perform highlight analysis on the video sample based on the first multimodal features to obtain highlight prediction information. The model training module is used to iteratively train the large language model based on the difference between the specular reference information and the specular prediction information; wherein, after completing the iterative training, the large language model is used as a specular fragment detection model.
[0008] According to another aspect of this disclosure, a video processing apparatus is also provided, the apparatus comprising: The video acquisition module is used to acquire the original video. The highlight detection module is used to extract second media materials of multiple media types from the original video based on a second prompt text using a highlight fragment detection model. The second media materials are then encoded to obtain second multimodal features, and the original video is analyzed based on these second multimodal features to obtain target highlight information. The highlight fragment detection model is trained using the aforementioned training method. The second prompt text is a task description text used to guide the highlight fragment detection model to perform highlight fragment detection based on the second media materials. The video editing module is used to perform editing operations on the original video based on the target highlight information to generate a target video corresponding to the original video.
[0009] According to another aspect of this disclosure, an electronic device is also provided, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the preceding descriptions.
[0010] According to another aspect of this disclosure, a computer-readable storage medium is also provided, the storage medium storing a computer program for performing the method described in any of the preceding claims.
[0011] According to another aspect of this disclosure, a storage medium is also provided for storing a target video obtained based on any of the video processing methods described above.
[0012] According to another aspect of this disclosure, a computer program product is also provided that, when run on a computer, causes the computer to perform the method as described in any of the preceding claims.
[0013] The technical solution provided in this disclosure has the following advantages compared with the prior art: The training method for the highlight fragment detection model provided in this disclosure acquires video samples labeled with highlight reference information as supervision signals. Based on a first prompt text, a large language model extracts first media materials of various media types from the video samples. This allows the large language model to focus on highlight fragment detection under unified task instructions, reducing reliance on manual annotation and subjective judgment. Furthermore, encoding the various types of first media materials effectively fuses information from different modalities, generating first multimodal features with comprehensive representation capabilities in a unified semantic space. This provides a rich and semantically consistent feature foundation for the structured prediction of highlight points. Subsequently, highlight analysis is performed on the video samples based on the first multimodal features to obtain highlight prediction information aligned with the highlight reference information structure. Compared to methods relying solely on visual information or human experience, this helps improve the accuracy and comprehensiveness of the large language model's semantic understanding of highlight fragments. Further, the large language model is iteratively trained based on the difference between the highlight reference information and the highlight prediction information. The trained large language model can then be used as a highlight fragment detection model in practical applications. Therefore, this training method can support the automated detection and extraction of highlight segments in subsequent video processing, effectively reducing manual intervention, shortening the production cycle of highlight videos, and reducing the quality fluctuations of finished products caused by individual subjective aesthetic differences, making large-scale, efficient, and quality-controllable video content distribution possible.
[0014] The video processing method provided in this disclosure, after acquiring the original video, extracts second media materials of various media types from the original video based on a second prompt text using a highlight fragment detection model. This highlight fragment detection model is obtained through the training method described above and can be used for automated detection and extraction of highlight fragments, laying a data foundation for subsequent accurate detection of highlight fragments. The second media materials are then encoded to obtain second multimodal features, and the original video is analyzed for highlight information based on these second multimodal features. The second prompt text is a task description text used to guide the highlight fragment detection model to perform highlight fragment detection based on the second media materials. Thus, the second prompt text, as a task instruction, effectively constrains the detection direction of the highlight fragment detection model, ensuring that the output target highlight fragment aligns with user intent or business requirements, thereby improving the accuracy of the target highlight fragment. Finally, the original video is edited based on the target highlight information to generate a target video corresponding to the original video, realizing the transformation from semantic understanding to actual content production.
[0015] Therefore, the training method described above provides a highlight detection model with highlight detection capabilities for video processing methods, which then directly apply this trained highlight detection model in video editing scenarios. The entire solution requires no manual intervention and can automatically generate high-quality target videos on demand with high efficiency, significantly improving the efficiency, flexibility, and task adaptability of video editing, and effectively meeting the needs of large-scale video content distribution. Attached Figure Description
[0016] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart of the training method for the highlight fragment detection model described in the embodiments of this disclosure; Figure 2 This is a flowchart of the feature enhancement method described in the embodiments of this disclosure; Figure 3 This is a flowchart of the video processing method described in the embodiments of this disclosure; Figure 4 This is a structural block diagram of the training device for the highlight fragment detection model described in the embodiments of this disclosure; Figure 5 This is a structural block diagram of the video processing apparatus described in the embodiments of this disclosure; Figure 6 This is a structural block diagram of the electronic device described in an embodiment of this disclosure. Detailed Implementation
[0019] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0020] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0021] Currently, manually generated highlight montage videos suffer from unstable quality, long production cycles, and low efficiency, failing to meet the demands of large-scale video content distribution. Therefore, this disclosure provides a training method for a highlight segment detection model and a video processing method. For ease of understanding, the embodiments of this disclosure are described below.
[0022] Figure 1 This flowchart illustrates a training method for a highlight fragment detection model provided in this embodiment. This method is applicable to model training, and the trained model is generally used in video processing scenarios such as highlight fragment detection and highlight fragment editing. The training method for the highlight fragment detection model can be executed by a training device for the model, which can be implemented using software and / or hardware, specifically, for example, an electronic device or a server. The electronic device can include communication-enabled devices such as in-vehicle hosts, tablets, desktop computers, laptops, and smartphones. The server can be a cloud server or server cluster, or other devices with storage and computing capabilities.
[0023] Reference Figure 1 The training method for the highlight fragment detection model provided in this embodiment may include the following steps S102 to S106.
[0024] S102, acquire video samples; wherein the video samples are marked with specular reference information; wherein the specular reference information is used to represent the information of specular segments used as a reference standard.
[0025] In this embodiment, video samples are used as input samples and specular reference information is used as output samples, together forming a training data pair. This training data pair is used to train a large language model, and the trained large language model is used as a specular detection model for performing specular detection tasks on videos.
[0026] Highlight segment detection refers to the task of automatically identifying, locating, and structurally outputting highlight information from video content using a highlight segment detection model. Highlight segments describe highlight segments that conform to the definition of highlights. Highlight segments can be understood as compelling video clips extracted from the original long video that possess visual value, high information density, and represent the core highlights of the video. Through highlight segments, the most attractive, representative, and communicative information from the original long video is conveyed in a shorter duration.
[0027] During the training phase, video samples are labeled with corresponding highlight reference information to guide the large language model in learning the detection and structured output of highlight points. The highlight reference information is used to represent the information of the highlight segment as a reference standard, which includes, but is not limited to: the start and end time of the highlight, the type of highlight, the character information of the film and television characters contained in the highlight segment, the title information, the plot summary, and the explanation of the reason for the highlight, which is the reason for identifying this video segment as a highlight segment.
[0028] Highlight reference information uses structured data to represent highlight location information, aiming to accurately and parsably describe the spatiotemporal location and semantic attributes of one or more highlight events in a video clip. Specifically, this highlight reference information may include: structured fields and corresponding highlight location description text. That is, each structured field corresponds to a type of highlight semantic dimension and is equipped with corresponding highlight location description text as its field value.
[0029] As an example, structured fields may include, but are not limited to: highlight start and end times, highlight type, character information, plot summary, title text, and explanation of the reason for the highlight. Among these: The highlight start and end times are used to identify the time boundaries of the highlight segment within the video clip; the corresponding highlight point description text is as follows: [t_start, t_end]).
[0030] Highlight type is used to classify the semantic category of highlight events; the corresponding highlight point description text is such as: fighting type, tear-jerking type, or emotional outburst type, etc.
[0031] Character information is used to identify key characters in the highlighted scenes; the corresponding highlight description text is such as: protagonist A, villain B.
[0032] A plot summary is used to briefly summarize the core plot of the highlight scene in natural language; the corresponding description of the highlight is as follows: The female lead C chases after the car in the rain to confess her love, reveals her three-year-old secret, and her emotional outburst triggers a key turning point.
[0033] Title text, used to generate video cover or recommended content; its corresponding highlight description text, such as: Confession in the rain, three years of hidden secrets finally revealed.
[0034] The explanation of the highlight is used to explain the basis for judging the segment as a highlight, supporting the interpretability of the large language model and user understanding; the corresponding highlight point description text is as follows: This segment focuses on the emotional climax of the character and the revelation of the core secret. It is a key node that drives the turning point of the main plot relationship and has strong dramatic tension and audience resonance.
[0035] This embodiment uses the aforementioned structured spectrophotometric reference information to annotate video samples, which not only provides fine-grained supervision signals for multiple prediction branches of the large language model, but also ensures that the large language output has good semantic consistency and compatibility with downstream applications, laying a data foundation for the automatic generation and interpretability analysis of spectrophoto montage.
[0036] S104: Based on the first prompt text, a large language model is used to extract first media materials of multiple media types from video samples. The first media materials are encoded to obtain first multimodal features. Based on the first multimodal features, the video samples are subjected to specular analysis to obtain specular prediction information.
[0037] In this embodiment, video samples are input into a large language model, which then performs a highlight segment detection task on the video samples. To better achieve task-oriented highlight segment reading and detection, this embodiment can simultaneously input a first prompt text as a task instruction into the large language model.
[0038] The first cue text can be a task description in natural language, used to guide the large language model to focus on the highlight fragment detection task during the training phase and to standardize its output format. As an example, the first cue text may include the following: task definition, highlight fragment selection criteria, and output format requirements.
[0039] The task definition is used to indicate the type of task that the large language model needs to perform, such as: "You are a professional highlight fragment detection assistant. Please identify and locate all key fragments that meet the highlight definition from the given video."
[0040] The highlight segment selection criteria are used to represent the general semantic definition or judgment standard of highlight segments, so as to unify the discrimination logic of the large language model on different video samples. For example, the highlight segment selection criteria include the following quantifiable judgment information in multiple dimensions: "(i) Dialogue emotional intensity dimension: when the density of emotional words contained in the first dialogue text corresponding to the video segment exceeds the preset emotional threshold, it can show the core emotional conflict of the character, and thus, the segment is regarded as a candidate highlight segment; (ii) Visual dynamic change dimension: when the change in the optical flow field between adjacent image frames in the video segment exceeds the preset dynamic threshold, it indicates that it contains key plot segments with high visual impact, and thus, the segment is regarded as a candidate highlight segment; (iii) Transition frequency dimension: when the density of video transition points in the video segment exceeds the preset transition density threshold, it indicates that it has a high probability of being a key transition, and thus, the segment is regarded as a candidate highlight segment." The above highlight segment selection criteria are used to instruct the large language model to select highlight segments that meet the conditions from the video samples based on at least one of the above dimensions.
[0041] The output format requirements are used to instruct the large language model to output in a specified structured format to ensure that the output specular prediction information can be compared with the structured specular reference information.
[0042] During training, the initial prompt text and video samples are used as input to the large language model.
[0043] The large language model used in this embodiment can mainly include the following core components: encoding module, feature enhancement module, and multiple prediction branches.
[0044] The encoding module is used to input video samples, extract first media materials of various media types from the video samples, encode each first media material, and output a multimodal feature set containing features of the various first media materials. The various media types may include at least two of the following: text, image, and audio; the encoding process includes at least encoding, feature concatenation, and feature enhancement.
[0045] The first media material can include at least the following data: the first video clip, the corresponding first dialogue text, the first video transition point, and the first audio clip. These first media materials from various media types are interconnected in the temporal dimension, collectively forming multimodal data with semantic consistency and structural context, providing the basic input for subsequent highlight point identification.
[0046] The following example illustrates how to extract first media materials of multiple media types from video samples.
[0047] Extract a first video segment of a preset time length from the video sample; the video sample is generally an original long video, such as a complete movie video; divide the video sample into multiple video segments according to the preset time length (such as 5 minutes), and take any one of the video segments as the first video segment.
[0048] To ensure accurate temporal alignment of multimodal data, this embodiment uses the global timeline of the first video segment extracted from the video sample, rather than employing local relative time. In other words, any point in time within the first video segment (e.g., frame 5, second 2.3, etc.) is directly mapped to an absolute timestamp in the video sample. Accordingly, all time-related information determined based on this first video segment, including but not limited to highlight start and end times, transition point positions, and dialogue alignment timestamps, is based on the global timeline of the video sample, strictly maintaining spatiotemporal consistency with the content of the video sample.
[0049] The above method can avoid multimodal misalignment caused by fragment-level local temporal offset, ensuring that the visual, text, and structural modalities are always under a unified temporal reference frame during subsequent fusion, training, and inference processes, thereby guaranteeing the accuracy of highlight location and the traceability of the edited video from the source video.
[0050] Adjacent videos of a first preset duration are extended before and after the timeline of the first video segment to form an enhanced video for dialogue extraction. Semantic recognition is performed on the textual and / or audio dialogue contained in the enhanced video. Based on the semantic recognition results, a semantically complete first dialogue text corresponding to the first video segment is determined. The first dialogue text is used to express the linguistic semantics synchronized with the visuals of the first video segment.
[0051] In practical processing, dialogue extracted solely from the first video segment may suffer from semantic incompleteness. This is because complete dialogue in the original video may span multiple consecutive frames, and when the video is sliced according to fixed durations or event boundaries, a complete line of dialogue may be split into two adjacent video segments. For example, the beginning of the first video segment may only contain the latter half of a line from the previous video segment, or the end may only contain the first half of a line from the subsequent video segment. This results in fragmented sentences, semantic breaks, or missing context in the extracted first dialogue text, affecting the accuracy of subsequent multimodal understanding and highlight point identification.
[0052] In this embodiment, based on the first video segment, an adjacent video segment of a first preset duration (e.g., 3 seconds) is extended before and after its timeline to form an enhanced video for dialogue extraction. This enhanced video can cover the first video segment and its context, ensuring that the complete sentence boundaries semantically related to the first video segment can be captured when performing speech dialogue recognition or text dialogue alignment. Subsequently, the original dialogue text is extracted from the enhanced video, and the portion of dialogue text that is temporally aligned with the first video segment is extracted as the final first dialogue text. In this way, the sentence segmentation problem caused by the inconsistency between the video segment boundaries and the speech sentence boundaries can be effectively avoided, significantly improving the completeness of the dialogue semantics and the contextual coherence, and providing high-quality dialogue modal input for multimodal feature fusion and highlight point detection.
[0053] The similarity value between any two adjacent image frames in the first video segment is calculated. When the similarity value between adjacent image frames is lower than a preset transition threshold, it indicates a significant abrupt change in the image content, which may correspond to a shot change, scene jump, or other forms of video transition. Therefore, the time point where the similarity value is lower than the transition threshold is marked as the first video transition point. The first video transition point is used to characterize the shot boundaries or editing structure information within the first video segment, serving as structural modality input in multimodal data to assist in the semantic understanding of subsequent highlight points and the logical reasoning of mixed editing.
[0054] A first audio segment aligned with the first video clip on the timeline is extracted from the video sample. To ensure the semantic integrity of the audio content, adjacent audio segments of a second preset duration are extended before and after the timeline of the first video clip to form an enhanced audio segment for audio feature extraction. This enhanced audio segment undergoes audio signal processing to extract acoustic features including volume energy spectrum, zero-crossing rate, spectral centroid, and fundamental frequency. The acoustic features of the aligned portion are then extracted based on the time boundary of the first video clip to serve as the audio feature representation of the first audio segment. These audio features can subsequently be used for cross-modal interaction with visual features and dialogue features in a large language model to assist in identifying highlight events based on sound intensity or prosodic changes, such as explosions, cheers, or background music climaxes.
[0055] The aforementioned primary media materials are interconnected in terms of time and semantics, collectively forming a complete set of multimedia data. The encoding module encodes each primary media material, outputting a multimodal feature set, which contains single-modal features of various primary media materials.
[0056] The feature enhancement module is used to input a multimodal feature set, perform feature interaction enhancement processing on the multimodal feature set based on the attention mechanism, and output the first multimodal feature with feature interaction.
[0057] The input to multiple prediction branches is the first multimodal feature; different prediction branches are used to perform different prediction tasks based on the first multimodal feature and output their respective prediction results. The prediction results of multiple prediction branches are used to generate specular prediction information in a structured data form.
[0058] In this embodiment, the encoding module of the large language model performs modal adaptation feature extraction on the first media materials, such as the first video clip, the first dialogue text, and the first video transition point, to obtain the single-modal features corresponding to each type of first media material. Subsequently, these single-modal features from different sources are concatenated to form a preliminary multimodal joint representation, i.e., a multimodal feature set.
[0059] To further enhance the semantic richness and cross-modal consistency of features, a multimodal feature set is input into the feature enhancement module. The feature enhancement module performs context-aware feature interaction and enhancement processing through a pre-defined attention mechanism to obtain the first multimodal feature. This first multimodal feature integrates multiple contextual information such as visual content, dialogue semantics, and transition structure, possessing high discriminative power and task versatility, and can serve as a unified input basis for subsequent multi-task prediction.
[0060] Next, the big prediction model generates specular prediction information for the video samples based on the first multimodal features.
[0061] In this embodiment, after generating the first multimodal feature, it is input into multiple prediction branches running in parallel within a large language model. These multiple prediction branches share the first multimodal feature as input, but each performs predictions targeting different semantic dimensions of the highlight points. Based on this unified representation of the first multimodal feature, the multiple prediction branches synchronously execute their respective prediction tasks, such as highlight temporal boundary regression, highlight type classification, character recognition, title generation, plot summary generation, and highlight cause generation.
[0062] The prediction results output from each prediction branch are integrated according to a predefined structured data format to generate complete specular prediction information. This specular prediction information is structurally consistent with the specular baseline information, including: structured fields and corresponding specular point description text.
[0063] S106, Based on the difference between the specular reference information and the specular prediction information, the large language model is iteratively trained; wherein, after completing the iterative training, the large language model is used as a specular fragment detection model.
[0064] This embodiment may include: calculating the loss value between the specular reference information and the specular prediction information according to multiple preset loss functions; wherein, the multiple loss functions correspond to multiple prediction branches in the large language model; then jointly training the large language model based on the weighted value of the multiple loss values until the total loss function value converges to the preset value, determining that the convergence condition has been met, and using the large language model as the specular fragment detection model.
[0065] This embodiment employs a multi-task joint training strategy to perform end-to-end optimization of a large language model in order to construct a highlight fragment detection model. Specifically, multiple loss functions are configured for each prediction branch in the large language model, each matching its prediction task, resulting in multiple loss functions. For example, smoothing L1 loss or mean squared error loss is used for time regression tasks, while cross-entropy loss is used for classification tasks.
[0066] During training, the loss values calculated by each prediction branch on the current batch of multimedia samples are weighted and summed according to preset weight coefficients to obtain the total loss function value. This weighting strategy can be statically set, such as manually assigning weights based on task importance; or it can be dynamically adjusted, such as automatically learning weights based on uncertainty estimation or gradient normalization. Subsequently, the model parameters of the large language model are synchronously updated through the backpropagation algorithm with the goal of minimizing the total loss function value.
[0067] Training iteratively continues until the total loss function value stably converges to a preset threshold on the validation set, or meets other preset convergence conditions, such as the loss decreasing less than the tolerance for several consecutive rounds. Once the convergence condition is met, the large language model is considered trained successfully, and the current state of the large language model is solidified as a highlight fragment detection model. The trained highlight fragment detection model has the ability to automatically identify highlight points from multimodal input and output structured highlight information, and can be directly deployed for applications such as intelligent video editing and content summarization.
[0068] To better understand, the following examples provide a more detailed description of the training method for the above-mentioned highlight fragment detection model.
[0069] In this embodiment, step S104, where the first media material is encoded using a large language model to obtain the first multimodal feature, may include: Step 1: The large language model encodes each type of first media material into its own single-modal features.
[0070] Taking the first media material, including the first video clip, the first dialogue text, the first video transition point, and the first audio clip, as an example, this embodiment can be: using a large language model, the first video clip, the first dialogue text, the first video transition point, and the first audio clip are respectively encoded into visual features, dialogue features, transition point features, and audio features.
[0071] In one implementation, the first video segment v is processed through the visual layer of a large language model. i Encode and output visual features S that represent the content, motion dynamics, and scene semantics of the image. i :S i =f(s) i ), where i represents the sequence number of the first video segment in the video sample.
[0072] The first dialogue text was processed by a text encoder based on a large language model. i Encode the dialogue and output the dialogue features L that contain semantic, emotional, and narrative information. i :L i =f(l i ).
[0073] The first video transition point c is obtained through the embedding layer of a large language model or a fully connected network. i Encode the transition point features C that characterize the video editing structure. i :C i =f(c i ).
[0074] The first audio segment a is processed by the audio encoder of the large language model. i Encode and output audio features A that represent sound intensity, prosodic variation, and the semantics of acoustic events. i A i =f(a i ).
[0075] Although the aforementioned visual features, dialogue features, transition point features, and audio features come from different sources, they are all processed through the same large language model f(·), which ensures feature space alignment and encodes them into feature matrices of the same dimension.
[0076] Step 2: Combine multiple single-modal features to form a multimodal feature set.
[0077] Following the example above, this step can be: concatenating visual features, dialogue features, transition point features, and audio features into a multimodal feature set. Referring to the following formula (1), the visual features, dialogue features, transition point features, and audio features corresponding to the first video segment are sorted in chronological order to form a shape R. d×T Multimodal feature set X i : (1) Among them, X i Let T represent the multimodal feature set, where T represents the sequence length of the first video segment, and d is the feature dimension. Visual features, dialogue features, transition point features, and audio features constitute independent multimodal feature sets, which together form the raw feature data for subsequent multimodal attention interactions.
[0078] Step 3: Perform feature interaction enhancement processing on the multimodal feature set based on the attention mechanism to obtain the first multimodal feature; the first multimodal feature is used to represent the interaction result of multiple single-modal features and has feature interactivity.
[0079] In one embodiment, the attention mechanism includes: a grouped query attention mechanism and a cross-modal attention mechanism. For example... Figure 2 As shown, feature interaction enhancement processing is performed on the multimodal feature set based on the attention mechanism to obtain the first multimodal feature, which may include: S202, according to the group query attention mechanism, the current feature is subjected to feature interaction enhancement processing with the multimodal feature set to obtain the first enhanced feature; wherein, the current feature is any one of multiple unimodal features, that is, any one of visual features, dialogue features, transition point features and unimodal features.
[0080] In this embodiment, based on the Group Query Attention (GQA) mechanism, the multimodal feature set is grouped according to modality, with each group corresponding to one modality. In other words, the multimodal feature set is divided into multiple groups corresponding to the first media material. In this example, the first media material, which corresponds one-to-one with the first video clip, the first dialogue text, the first video transition point, and the first audio clip, can form the following four independent feature groups: Group 1 is the group of visual features, Group 2 is the group of dialogue features, Group 3 is the group of transition point features, and Group 4 is the group of audio features.
[0081] Each group is independently generated with a unique query vector, which interacts with a key and value constructed from a unified set of multimodal features (representing all modal features). This effectively establishes the dependency relationship between each modality and the global multimodal context while preserving the local semantic characteristics of each modality.
[0082] For ease of description, visual features, dialogue features, transition point features, and audio features are referred to as current features. "Current feature" is a general term used to refer to any one of the four unimodal features, and can be represented as... , where g represents the group sequence number of the single-modal feature, that is, the g-th group of single-modal features.
[0083] In this case, this embodiment uses a grouped query attention mechanism to perform feature interaction enhancement processing on the current feature and the multimodal feature set to obtain the first enhanced feature, which may include: A query vector is generated for the current feature based on the query projection matrix. Specifically, the query vector for the current feature is: .in, For current features; A pre-defined query projection matrix for the current feature is used to ensure that different modal features initiate attention requests based on their own characteristics; This is the query vector for the current feature.
[0084] For example, the query vector for visual features is The query vector for dialogue features is The query vector for transition point features is .
[0085] Keys and values are generated for the multimodal feature set based on the key projection matrix and value projection matrix, respectively. Specifically, the key is: The value is ; where X i W is a multimodal feature set. k W is the preset key projection matrix. V The projection matrix is a preset value, K i As a key, V i The value is...
[0086] Based on the grouped query attention mechanism, the query vector and the key and value are subjected to feature interaction enhancement processing to obtain the current enhanced features.
[0087] Specifically, the enhanced features can be represented by the following formula (2): (2) in, This represents the enhanced features of the current feature in group g, where G is the number of groups into which the multimodal feature set is divided. In this example, there are four groups: visual features, dialogue features, transition point features, and audio features, so G=4.
[0088] Then, the enhanced features from multiple single-modal features are concatenated to generate the first enhanced feature. Specifically, the enhanced features from visual features, dialogue features, transition point features, and audio features are concatenated to generate the first enhanced feature.
[0089] The first enhancement feature can be referred to as shown in the following formula (3): (3) Among them, Wo This represents the preset fusion matrix. This represents the enhanced features of the single-modal features from group 1 to group G. This indicates the first enhancement feature.
[0090] This embodiment employs a grouped query attention mechanism. Each query group, while retaining its own local perspective within its single-modal features (e.g., visual focus on action changes, dialogue focus on emotional expression), can also perceive and aggregate global contextual information from all single-modal features. For example, the query vector for visual features not only focuses on the visual content but also dynamically adjusts its attention distribution based on the emotional intensity of dialogue, the narrative position of transition points, and the background music in the audio. Therefore, this embodiment effectively achieves a balance between local specificity and global synergy, significantly improving the ability of large language models to understand complex multimodal highlight events.
[0091] S204. Based on the cross-modal attention mechanism, at least two single-modal features are subjected to feature interaction enhancement processing to obtain a second enhanced feature.
[0092] To fully explore the complementarity and synergy among multiple unimodal features, this embodiment adopts a cross-modal attention mechanism (CMA) to perform pairwise or multimodal interaction enhancement processing on at least two unimodal features among visual features, dialogue features, transition point features and audio features to generate a second enhanced feature.
[0093] Unlike grouped query attention mechanisms where each group of queries interacts with a global key-value pair, cross-modal attention mechanisms focus on establishing targeted semantic bridges between specific modalities. Specifically, one type of unimodal feature is used as the query, and another type or more unimodal features are used as keys and values. Through attention computation, the query modality retrieves and aggregates semantically relevant contextual information from the target modality, thereby achieving information supplementation and feature enhancement between unimodal features.
[0094] As an example, at least two unimodal features include visual features and dialogue features. In this embodiment, to enhance the fine-grained alignment between visual content and linguistic semantics, a cross-modal attention mechanism is used to interactively enhance the visual features and dialogue features, generating a second enhanced feature.
[0095] Specifically, visual features are used as the query vector Q. S This is used to characterize the semantic focus of the first video segment; simultaneously, the dialogue features are used as keys K. L Sum V L This constitutes a semantic memory bank for dialogue. The second enhancement feature is determined according to the following formula (4). .
[0096] (4) This second enhancement feature not only preserves the original visual representation, but also explicitly injects the dialogue information most relevant to its semantics, thereby effectively improving the ability of the large language model to recognize visual and linguistic highlight events (such as emotional confessions, key lines combined with close-up shots, etc.), and providing stronger multimodal semantic support for the accurate positioning and structured description of subsequent highlight points.
[0097] As another example, at least two unimodal features include visual features, dialogue features, and transition point features. In this embodiment, a cross-modal attention mechanism is used to interactively enhance the visual features, dialogue features, and transition point features to generate a second enhanced feature.
[0098] Specifically, visual features are used as the query vector Q. s Simultaneously, dialogue features and transition point features are used as key K. L K C Sum V L V C Together, they constitute a multimodal semantic memory. The second enhancement feature is determined using the following formula (5). .
[0099] (5) in, This means that visual features are used to retrieve language information related to the semantics of the current scene from the features of the dialogue, thereby achieving alignment enhancement between visual content and dialogue semantics, such as enhancing the understanding of the semantics of the characters' dialogue in close-up shots.
[0100] This means that visual features are used to retrieve editing structure information related to the rhythm of the current scene from the transition point features, enabling the large language model to perceive the relationship between scene switching and narrative rhythm, such as enhancing the recognition of action-intensive scenes in fast transition segments.
[0101] Through the aforementioned dual-path cross-modal attention computation, the large language model can simultaneously supplement and enhance visual features from two dimensions: dialogue semantics and transition structure. This generates a second enhanced feature that integrates multimodal semantics, providing more comprehensive semantic support for the accurate localization of subsequent highlight points. The cross-modal attention mechanism effectively aligns visual content with linguistic semantics (e.g., the correspondence between close-up shots and emotional dialogue), visual content with acoustic signals (e.g., the synchronization of action intensity with background music climaxes), and structural information with narrative logic (e.g., the connection between transition rhythm and plot progression).
[0102] Finally, the enhanced features resulting from cross-modal interactions among multiple unimodal features are fused to generate a second enhanced feature that is semantically rich and tightly aligned across modalities, providing stronger multimodal semantic support for the accurate localization and structured description of subsequent highlight points.
[0103] S206, predict the current weight based on multiple single-modal features; wherein, the current weight is used to represent the degree of dependence on the first and second enhancement features.
[0104] In order to achieve adaptive fusion of different feature enhancement paths, this embodiment can first concatenate multiple single-modal features into a contextual representation feature, which fully preserves the information of each modal feature.
[0105] Subsequently, the current weights are predicted from the context representation features using a pre-defined weight matrix and activation function. Specifically, the context representation features are input into a lightweight weight prediction subnetwork, which may include a learnable weight matrix. and nonlinear activation functions The current weights are shown in the following example: (6) in, As the current weight, To construct a contextual representation feature by concatenating visual features, dialogue features, transition point features, and audio features, This is the weight matrix. This is the activation function.
[0106] Current weight These are dynamic weights used to characterize the relative reliability or contribution of the grouped query attention mechanism and cross-modal attention in the first video segment. The current weights adapt to the input content: when the dialogue in the first video segment is rich and highly relevant to the visuals, the current weights tend to be smaller, indicating that cross-modal attention is more reliable; when visuals or structural information dominates (such as action scenes without dialogue), the current weights tend to be larger, emphasizing the importance of the grouped query attention mechanism based on global context modeling.
[0107] S208, the first enhanced feature and the second enhanced feature are fused according to the current weights to obtain the first multimodal feature.
[0108] In this embodiment, a dynamic current weight that can represent the degree of dependence on the first and second enhanced features is introduced. Referring to the following formula (7), the first enhanced feature is weighted and fused according to the current weight. Second Enhancement Features The first multimodal feature is obtained. : (7) This embodiment uses dynamic current weights to fuse the first and second enhanced features, enabling the large language model to adaptively adjust the attention output according to modal importance, automatically decide when to trust which attention mechanism, significantly improve the robustness and semantic accuracy of highlight point recognition, and effectively achieve better feature combination.
[0109] In the above embodiments, by encoding the first media materials of each modality separately and concatenating them into a multimodal feature set, the specificity of visual, dialogue, transition points, and audio modalities is effectively preserved. Based on this, an attention mechanism is introduced to enhance the interaction of the multimodal feature set, enabling various single-modal features to fully perceive the global context while retaining their own information, significantly improving the semantic richness and cross-modal consistency of the features. The resulting first multimodal feature more accurately characterizes the multidimensional attributes of highlight events, laying a solid foundation for subsequent high-precision, interpretable highlight point recognition and structured output.
[0110] In this embodiment, step S104, which involves performing specular analysis on the video samples based on the first multimodal features to obtain specular prediction information, includes: The first multimodal feature is input into multiple prediction branches in the large language model; wherein the prediction branches include at least: highlight time branch, highlight type branch, role branch and title branch.
[0111] The start and end times of highlight segments in the first video clip are determined using a highlight time branch. Specifically, the first multimodal feature is input into the highlight time branch, which simultaneously encodes the time-location guidance text in the first prompt text into a time guidance vector. During the training phase, the time-location guidance text in the first prompt text is described using natural language, for example: "Please accurately locate the start and end frames of each highlight segment; the boundaries should fall at shot transition points or dialogue pauses."
[0112] High-energy point temporal regression analysis of the first multimodal feature is performed using the time-guided vector through the highlight time branch: This outputs the start and end times of the highlight segments in the first video clip, which may include a start timestamp and an end timestamp.
[0113] While outputting the start and end timestamps, the highlight time branch generates a corresponding confidence score for each timestamp, representing the reliability of the start and end times. This confidence score is supervised and optimized during the training phase using a boundary regression loss function.
[0114] The start and end times of the above-mentioned highlight time branch output serve as the time dimension field in the highlight prediction information, providing accurate time boundary basis for subsequent video editing operations.
[0115] The highlight type branch detects the highlight type corresponding to the highlight segment with a highlight score. Specifically, the first multimodal feature is input into the highlight type branch, and the highlight type branch encodes the guidance text about type classification in the first prompt text into a classification guidance vector. In the training phase, the guidance text about type classification in the first prompt text is, for example: "Please classify the highlight segment into the following types: emotional outburst, climax of the fight, and key turning point."
[0116] High-energy point multi-class classification is performed on the first multimodal feature using the high-light type branch based on the classification guide vector: This outputs the highlight type of the highlight fragment and its corresponding highlight score, and selects the type with the highest highlight score as the final output highlight type with the highest highlight score.
[0117] The highlight type and corresponding highlight score output by the above highlight type branch serve as the type dimension field in the highlight prediction information, providing a category basis for subsequent editing operations such as adding video tags and matching visualization templates to the original highlight clips.
[0118] The role information of characters in the highlight clip is identified through a role branch. Specifically, the first multimodal feature is input into the role branch, and the role branch encodes the guidance text about role identification in the first prompt text into a role guidance vector. The guidance text about role identification in the first prompt text is, for example: "Please identify all the key characters appearing in the highlight clip and label their roles as protagonist, supporting character, or villain. Key characters are those that meet at least one of the following conditions: (i) the cumulative on-screen time in the video sample exceeds a first preset duration threshold; (ii) the number of times the lines appear in the video sample exceeds a first preset frequency threshold; (iii) there is direct interaction (such as dialogue, confrontation, or cooperation) with the main characters already labeled in the video sample."
[0119] Character classification is performed based on the first multimodal feature using the character-guided vector through character branching: This allows for the identification of key figures and their roles within the highlighted segments, and the output of corresponding character information.
[0120] The character information output by the aforementioned character branch serves as the character dimension field in the highlight prediction information, providing semantic basis for subsequent editing operations such as adding character tags to the original highlight clips and generating character-specific highlight reels.
[0121] The title information for highlight segments is generated through a title branch. Specifically, the first multimodal feature is input into the title branch, which simultaneously encodes the prompt text about title generation in the first prompt text into a title guidance vector. During the training phase, the prompt text about title generation in the first prompt text might be, for example, "Please generate a short title for each highlight segment, with a title length between 8 and 15 characters."
[0122] Text generation is performed using the title branch based on the title-guided vector and the first multimodal feature: This generates a concise and eye-catching title that summarizes the key points of the highlight segment, such as: "Confession in the rain! The secret of three years is finally revealed."
[0123] To enhance the diversity and quality of headline generation, a bundle search decoding strategy and a duplication penalty mechanism are introduced in the headline branch. The bundle search strategy retains multiple candidate headline sequences during the decoding process. By calculating the cumulative probability and length-normalized score of each sequence, the sequence with the highest comprehensive score is selected as the final headline output, effectively avoiding the local optimum problem caused by greedy decoding.
[0124] The repetition penalty mechanism adds a score penalty to the generated word units to prevent repeated words or redundant expressions in the title, ensuring that the title is concise and refined.
[0125] The title information output by the title branch above serves as the title dimension field in the highlight prediction information, providing text material for subsequent editing operations such as adding dynamic title subtitles to the original highlight clips and generating video cover text.
[0126] Subsequently, highlight prediction information is generated based on the start and end times of the highlight fragment, highlight type, character information, and title information.
[0127] In another embodiment, the prediction branch may further include at least one of the following: a plot branch and a cause branch; correspondingly, this embodiment also includes at least one of the following: Plot summaries of highlight segments are extracted through plot branches. Specifically, based on the guiding text about plot analysis in the initial prompts, semantic understanding and summary generation are performed on the first multimodal features through plot branches: This outputs a natural language plot summary that accurately summarizes the core plot of the highlight scenes, such as: "The female lead chases after the car in the rain to confess her love, revealing a three-year-old secret, and her emotional outburst triggers a key turning point."
[0128] The explanation of the highlight cause for the highlight fragment is generated through causal branching. Specifically, attribution analysis is performed based on the introductory text about cause analysis in the first prompt and the first multimodal features. This generates a natural language explanation of why the segment was identified as a highlight, such as: "This segment vividly portrays the emotional climax of the character and the revelation of the core secret. It is a key node that drives the turning point of the main plot and has strong dramatic tension and audience resonance."
[0129] The highlight prediction information is generated by taking the start and end times of the highlight segment, highlight type, character information, title information, plot summary, and explanation of the reason for the highlight.
[0130] After multiple prediction branches complete multi-task predictions, the outputs of each prediction branch are integrated according to a predefined structured data format to generate the final specular prediction information.
[0131] Based on the above embodiments, during the training process of the large language model, multiple prediction branches are integrated and participate in the model training together. Based on the multiple loss functions corresponding to the multiple prediction branches, the large language model is jointly trained until the total loss function value converges to a preset value. When the convergence condition is met, the large language model is used as the highlight fragment detection model.
[0132] During training, the attention mechanism enhances cross-modal information interaction, captures local and global dependencies through grouped query attention, and the adaptive current weights can improve the ability of the large language model to identify key high-energy points.
[0133] The training method for the highlight fragment detection model provided in the above embodiments uses video samples labeled with highlight reference information as supervision signals. Based on the first prompt text, a large language model extracts first media materials of various media types from the video samples. This allows the large language model to focus on highlight fragment detection under unified task instructions, reducing reliance on manual annotation and subjective judgment. Furthermore, encoding these various types of first media materials effectively integrates information from different modalities such as visual, dialogue, transition points, and audio, generating first multimodal features with comprehensive representation capabilities in a unified semantic space. This provides a rich and semantically consistent feature foundation for the structured prediction of highlight points. Subsequently, highlight analysis is performed on the video samples based on the first multimodal features to obtain highlight prediction information aligned with the highlight reference information structure. Compared to methods that solely rely on visual information or human experience, this approach helps improve the accuracy and comprehensiveness of the large language model's semantic understanding of highlight fragments. Furthermore, the large language model is iteratively trained based on the difference between the specular baseline information and the specular prediction information. This allows the large language model to gradually learn the mapping ability from multimodal input to structured specular output. After training, the large language model can be used as a specular fragment detection model in practical applications. Therefore, this training method can support the automated detection and extraction of specular fragments in subsequent video processing, effectively reducing manual intervention, shortening the production cycle of specular videos, and reducing the quality fluctuations caused by individual subjective aesthetic differences. This makes large-scale, efficient, and quality-controllable video content distribution possible.
[0134] Therefore, the training method of this scheme makes full use of the highlight reference information of real highlights in video samples and the first media materials of various media types, which effectively improves the efficiency and stability of model training. The final trained highlight fragment detection model not only has accurate highlight point detection capability, but also outputs semantically complete and highly interpretable results, which can be stably applied to intelligent editing and content understanding tasks in video scenarios.
[0135] Figure 4 This is a flowchart illustrating a video processing method provided in an embodiment of the present disclosure. This method is applicable to video editing scenarios. The video processing method can be executed by a video processing device, which can be implemented using software and / or hardware, specifically such as an electronic device or a server. The electronic device can include devices with storage and computing capabilities, such as tablet computers, desktop computers, laptop computers, and smartphones. The server can be a cloud server or a server cluster, or other devices with storage and computing capabilities.
[0136] like Figure 4 As shown, the video processing method provided in this embodiment may include the following steps.
[0137] S402, retrieve the original video.
[0138] As examples, raw videos, such as a 90-minute film or television series or a 2-hour sports event recording, can be obtained from a video database or a user upload interface.
[0139] S404, using a highlight fragment detection model based on the second prompt text, extracts second media materials of multiple media types from the original video, encodes the second media materials to obtain second multimodal features, and performs highlight analysis on the original video based on the second multimodal features to obtain target highlight information; wherein, the highlight fragment detection model is trained according to the training method of the highlight fragment detection model provided in the aforementioned embodiment; the second prompt text is a task description text used to guide the highlight fragment detection model to perform the highlight fragment detection task based on the second media materials.
[0140] In this embodiment, a highlight fragment detection model is used to extract secondary media materials of various media types from the original video based on the second prompt text. This includes: Based on the second prompt text, the original video is extracted using a highlight fragment detection model. The extraction operation includes: dividing the original video into multiple second video segments according to a preset time length; extracting the second dialogue text, second video transition points, and second audio segments corresponding to each second video segment; and identifying each second video segment and its corresponding second dialogue text, second video transition points, and second audio segments as second media materials.
[0141] The highlight fragment detection model encodes each second media material based on the second prompt text to obtain second multimodal features, and generates the target highlight information of the original video based on the second multimodal features.
[0142] Specifically, in order to better achieve task-oriented highlight fragment detection, this embodiment can simultaneously input a second prompt text into the highlight fragment detection model as a task instruction to guide the highlight fragment detection model to focus on a specific type of highlight.
[0143] The second prompt text can be a task description text in natural language form, used to clarify information such as the definition of highlights, screening criteria, and application scenarios, thereby regulating the attention distribution and prediction preferences of the highlight fragment detection model.
[0144] Specifically, the second prompt text can be a natural language instruction entered by the user through the interface. This second prompt text can clearly reflect the user's creative intention, such as: "Please select all the highlight scenes that show the character's inner struggle, especially those containing silence, close-ups, or rain scenes."
[0145] Alternatively, first determine the video type of the original video, and then select a prompt text from the preset template library that matches the video type. For example, if the original video is a romance drama, the following prompt text would be matched: "Select highlight clips containing emotional climaxes such as confessions, misunderstandings, and reunions." If the original video is a football match, the following prompt text would be matched: "Prioritize goals, saves, red cards, and celebration moments."
[0146] During the inference process, the highlight fragment detection model jointly encodes the second cue text and the second media material, so that its multiple prediction branches output the meaning expressed by the cue text under unified semantic constraints. Figure 1 The target highlight information should include at least the following: target start and end time, target highlight type, target character information, target title information, target plot summary, and explanation of the reason for the target highlight.
[0147] This embodiment introduces a second prompt text guidance mechanism, enabling the highlight fragment detection model to flexibly adapt to diverse highlight definition requirements without modifying the network structure or retraining the model. During highlight fragment selection, the description of highlights in the second prompt text guides the model to select accurate highlight locations according to the description, and provides sufficient reasons to support the selection of the correct highlight fragment. This second prompt text serves as a task-level prior injected into the highlight fragment detection model's inference process, guiding it to dynamically adjust its attention focus and prediction preferences in the multimodal feature space. Based on this, the controllability, relevance, and business applicability of the detection results are significantly improved.
[0148] Therefore, by using the highlight fragment detection model to output target highlight information under the guidance of the second prompt text, the controllability of highlight fragment detection results, the high correlation between the output target highlight information and the task intent, and the business applicability are improved. It is applicable to multiple scenarios such as publicity, highlights, and review, and realizes the transformation from general highlight recognition to intent-driven intelligent highlight generation.
[0149] S406, Perform editing operations on the original video based on the target highlight information to generate the target video corresponding to the original video.
[0150] This embodiment may include: (1) editing multiple original highlight segments corresponding to the start and end times of multiple targets in the original video.
[0151] (2) Based on the target highlight type, target character information and target title information, add video tags and / or visualization materials to the original highlight clip to obtain the target highlight clip.
[0152] Specifically, based on the target highlight type (e.g., emotional confession), target character information (e.g., female lead A, male lead B), and target title information (e.g., "Confession in the Rain! Three-Year Secret Revealed"), video tags and / or visual materials are automatically added to each original highlight clip. Examples include: overlaying character name tags in the corner of the screen, embedding dynamic title subtitles at the beginning, or matching preset visual templates according to the highlight type, such as using a flashing border for a reversal type and adding a soft light filter for a heartwarming type, thereby improving the information delivery efficiency and viewing experience of the final product.
[0153] (3) Perform video editing and sorting on the target highlight segment to obtain the target video.
[0154] In a specific embodiment, if the target highlight information also includes an explanation of the cause of the target highlight and a summary of the target plot, the target highlight segments can be edited and sorted based on the explanation of the cause of the target highlight and the summary of the target plot to obtain the target video.
[0155] Editing multiple target highlight clips is not simply a matter of splicing them together, as this can easily lead to temporal inconsistencies and logical breaks, such as showing the ending before the cause. Therefore, in this embodiment, the explanation of the target highlight's cause and a summary of the target's plot can be used as key context, inputting them into a pre-trained generative model. This generative model is a pre-trained model used to edit multiple video clips to generate a new video.
[0156] For each target highlight segment, the generative model analyzes the narrative logic of the target highlight segment in the original video based on the target highlight reason explanation and target plot summary corresponding to the current target highlight segment, and determines the temporal information of the target highlight segment; the temporal information is used to represent the development progress of the story in the original video, such as the beginning, development, climax and ending of the story.
[0157] For example: if the timing information is a highlight at the beginning of the story, then place the target highlight segment at the beginning of the final film; if the timing information is a highlight of escalating conflict, then place the target highlight segment in the middle of the final film; if the timing information is a highlight of the ending, then place the target highlight segment at the end of the final film.
[0158] In addition, the timing information mentioned above can be optimized by combining information such as the second video transition point.
[0159] Then, based on the time sequence information, video editing and sorting are performed on each target highlight segment to ensure that the final output target video is coherent in terms of storyline, progressive in terms of emotion, and complete in terms of structure, avoiding the problem of highlights but chaos.
[0160] Through the above embodiments, not only can the automatic extraction of target highlight segments be achieved, but also the end-to-end generation of high-quality target videos with semantic structure, visual enhancement and narrative logic can be completed from the original video, significantly improving the practicality and user experience of intelligent editing.
[0161] The above embodiments provide a video processing method, including: first, acquiring an original video; second, extracting second media materials of various media types from the original video based on a second prompt text using a highlight fragment detection model, thereby enabling the highlight fragment detection model to analyze in structured and processable units, laying a data foundation for subsequent accurate detection of highlight fragments; then, encoding the second media materials to obtain second multimodal features, and performing highlight parsing on the original video based on the second multimodal features to obtain target highlight information; wherein, the second prompt text is a task description text used to guide the highlight fragment detection model to perform highlight fragment detection tasks based on the second media materials; thus, the second prompt text, as a task instruction, can effectively constrain the detection direction of the highlight fragment detection model, ensuring that the output target highlight fragment is aligned with user intent or business requirements, and improving the accuracy of the target highlight fragment. Finally, performing editing operations on the original video based on the target highlight information to generate a target video corresponding to the original video, realizing the transformation from semantic understanding to actual content production.
[0162] Therefore, this solution does not rely on manual labor and can automatically generate high-quality target videos on demand with high efficiency, significantly improving the efficiency, flexibility and task adaptability of video editing, and can better meet the needs of large-scale video content distribution.
[0163] Reference Figure 4 This disclosure provides a training apparatus for a specular fragment detection model, the apparatus comprising: The sample acquisition module 402 is used to acquire video samples; wherein the video samples are marked with highlight reference information; wherein the highlight reference information is used to represent information of highlight segments as a reference standard; The highlight prediction module 404 is used to extract first media materials of multiple media types from the video sample based on the first prompt text using a large language model, encode the first media materials to obtain first multimodal features, and perform highlight analysis on the video sample based on the first multimodal features to obtain highlight prediction information. The model training module 406 is used to iteratively train the large language model based on the difference between the specular reference information and the specular prediction information; wherein, after completing the iterative training, the large language model is used as a specular fragment detection model.
[0164] In one embodiment, the highlight prediction module 404 is further configured to: The large language model encodes each type of the first media material into its own single-modal features; The various single-modal features are combined to form a multimodal feature set; The first multimodal feature is obtained by performing feature interaction enhancement processing on the multimodal feature set based on the attention mechanism.
[0165] In one embodiment, the attention mechanism includes: a group query attention mechanism and a cross-modal attention mechanism; the specular prediction module 404 is further configured to: Based on the grouped query attention mechanism, the current feature is subjected to feature interaction enhancement processing with the multimodal feature set to obtain the first enhanced feature; wherein, the current feature is any one of the various single-modal features; Based on the cross-modal attention mechanism, at least two of the single-modal features are subjected to feature interaction enhancement processing to obtain a second enhanced feature; The current weight is predicted based on multiple single-modal features; wherein the current weight is used to represent the degree of dependence on the first enhancement feature and the second enhancement feature; The first enhanced feature and the second enhanced feature are fused according to the current weights to obtain the first multimodal feature.
[0166] In one embodiment, the highlight prediction module 404 is further configured to: Generate a query vector for the current feature based on the query projection matrix; Generate keys and values for the multimodal feature set based on the key projection matrix and value projection matrix, respectively; Based on the grouped query attention mechanism, the query vector is subjected to feature interaction enhancement processing with the key and the value to obtain the current enhanced feature; The enhanced features of the various single-modal features are concatenated to generate the first enhanced feature.
[0167] In one embodiment, the highlight prediction module 404 is further configured to: The various single-modal features are concatenated into a contextual representation feature; The current weights are predicted based on the context representation features using a preset weight matrix and activation function.
[0168] In one embodiment, the highlight prediction module 404 is further configured to: The first multimodal feature is input into multiple prediction branches in the large language model; wherein, the prediction branches include at least: a highlight time branch, a highlight type branch, a character branch, and a title branch; The start and end times of highlight segments in the video sample are determined by the highlight time branch. Detect the highlight type with highlight score corresponding to the highlight fragment by highlight type branch; The character information of the characters in the highlight clips is identified through the character branch; The title information of the highlight clip is generated through the title branch; Based on the start and end times, highlight type, character information, and title information of the highlight fragment, highlight prediction information is generated.
[0169] In one embodiment, the prediction branch further includes at least one of the following: a plot branch and a cause branch; the highlight prediction module 404 is also used for at least one of the following: Extract plot summaries of highlight segments from the aforementioned plot branches; The explanation of the highlight cause for the highlight fragment is generated through the aforementioned cause branch.
[0170] In one embodiment, the model training module 406 is further configured to: The loss value between the specular reference information and the specular prediction information is calculated based on multiple preset loss functions; wherein, the multiple loss functions correspond to multiple prediction branches in the large language model; The large language model is jointly trained based on the weighted values of multiple loss values until the total loss function value converges to a preset value. When the convergence condition is met, the large language model is used as the highlight fragment detection model.
[0171] Reference Figure 5 This disclosure provides a video processing apparatus, the apparatus comprising: Video acquisition module 502 is used to acquire the original video; The highlight detection module 504 is used to extract second media materials of multiple media types from the original video based on a second prompt text using a highlight fragment detection model, encode the second media materials to obtain second multimodal features, and perform highlight analysis on the original video based on the second multimodal features to obtain target highlight information; wherein, the highlight fragment detection model is trained according to the aforementioned training method for highlight fragment detection models; the second prompt text is a task description text used to guide the highlight fragment detection model to perform highlight fragment detection tasks based on the second media materials; The video editing module 506 is used to perform editing operations on the original video based on the target highlight information to generate a target video corresponding to the original video.
[0172] In one embodiment, the highlight detection module 504 is further configured to: Based on the second prompt text, the original video is extracted using a highlight fragment detection model; wherein, the extraction operation includes: dividing the original video into multiple second video segments according to a preset time length; and extracting the second dialogue text, second video transition points, and second audio segments corresponding to each second video segment; Each of the second video segments and its corresponding second dialogue text, second video transition points, and second audio segments are identified as the second media material.
[0173] In one embodiment, the target highlight information includes at least: target start and end time, target highlight type, target character information, and target title information; the video editing module 506 is further configured to: Edit multiple original highlight clips corresponding to the start and end times of the target from the original video; Based on the target highlight type, the target character information, and the target title information, video tags and / or visual materials are added to the original highlight clip to obtain the target highlight clip; The target highlight segment is processed and sorted to obtain the target video.
[0174] The device provided in this embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.
[0175] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 6 As shown, the electronic device 600 includes one or more processors 601 and memory 602.
[0176] The processor 601 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 600 to perform desired functions.
[0177] The memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may execute the program instructions to implement the training method and video processing method of the highlight fragment detection model of the embodiments of this disclosure described above, and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.
[0178] In one example, the electronic device 600 may also include an input device 603 and an output device 604, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0179] In addition, the input device 603 may also include, for example, a keyboard, a mouse, etc.
[0180] The output device 604 can output various information to the outside, including determined distance information, direction information, etc. The output device 604 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0181] Of course, for the sake of simplicity, Figure 6 Only some of the components of the electronic device 600 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 600 may include any other suitable components depending on the specific application.
[0182] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program for executing the training method and video processing method of the above-described highlight fragment detection model.
[0183] The present disclosure provides a computer program product for a training method for a highlight fragment detection model, a video processing method, an apparatus, an electronic device, and a medium, including a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.
[0184] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0185] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A training method for a highlight fragment detection model, characterized in that, The method includes: Acquire video samples; wherein the video samples are marked with specular reference information; wherein the specular reference information is used to represent information of specular segments serving as a reference standard; Based on the first prompt text, a large language model is used to extract first media materials of multiple media types from the video samples. The first media materials are encoded to obtain first multimodal features. Highlight analysis is performed on the video samples based on the first multimodal features to obtain highlight prediction information. Based on the difference between the specular reference information and the specular prediction information, the large language model is iteratively trained; wherein, after completing the iterative training, the large language model is used as a specular fragment detection model.
2. The method according to claim 1, characterized in that, The process of encoding the first media material to obtain the first multimodal feature includes: The large language model encodes each type of the first media material into its own single-modal features; The various single-modal features are combined to form a multimodal feature set; The first multimodal feature is obtained by performing feature interaction enhancement processing on the multimodal feature set based on the attention mechanism.
3. The method according to claim 2, characterized in that, The attention mechanism includes: a grouped query attention mechanism and a cross-modal attention mechanism; the feature interaction enhancement processing of the multimodal feature set based on the attention mechanism to obtain the first multimodal feature includes: Based on the grouped query attention mechanism, the current feature is subjected to feature interaction enhancement processing with the multimodal feature set to obtain the first enhanced feature; wherein, the current feature is any one of the various single-modal features; Based on the cross-modal attention mechanism, at least two of the single-modal features are subjected to feature interaction enhancement processing to obtain a second enhanced feature; The current weight is predicted based on multiple single-modal features; wherein the current weight is used to represent the degree of dependence on the first enhancement feature and the second enhancement feature; The first enhanced feature and the second enhanced feature are fused according to the current weights to obtain the first multimodal feature.
4. The method according to claim 3, characterized in that, The step of performing feature interaction enhancement processing between the current feature and the multimodal feature set according to the grouped query attention mechanism to obtain the first enhanced feature includes: Generate a query vector for the current feature based on the query projection matrix; Generate keys and values for the multimodal feature set based on the key projection matrix and value projection matrix, respectively; Based on the grouped query attention mechanism, the query vector is subjected to feature interaction enhancement processing with the key and the value to obtain the current enhanced feature; The enhanced features of the various single-modal features are concatenated to generate the first enhanced feature.
5. The method according to claim 3, characterized in that, The prediction of the current weight based on multiple single-modal features includes: The various single-modal features are concatenated into a contextual representation feature; The current weights are predicted based on the context representation features using a preset weight matrix and activation function.
6. The method according to claim 1, characterized in that, The step of performing specular analysis on the video samples based on the first multimodal features to obtain specular prediction information includes: The first multimodal feature is input into multiple prediction branches in the large language model; wherein, the prediction branches include at least: a highlight time branch, a highlight type branch, a character branch, and a title branch; The start and end times of highlight segments in the video sample are determined by the highlight time branch. Detect the highlight type with highlight score corresponding to the highlight fragment by highlight type branch; The character information of the characters in the highlight clips is identified through the character branch; The title information of the highlight clip is generated through the title branch; Based on the start and end times, highlight type, character information, and title information of the highlight fragment, highlight prediction information is generated.
7. The method according to claim 1, characterized in that, The prediction branch further includes at least one of the following: plot branch and cause branch; the method further includes at least one of the following: Extract plot summaries of highlight segments from the aforementioned plot branches; The explanation of the highlight cause for the highlight fragment is generated through the aforementioned cause branch.
8. The method according to claim 1, characterized in that, The step of iteratively training the large language model based on the difference between the specular reference information and the specular prediction information includes: The loss value between the specular reference information and the specular prediction information is calculated based on multiple preset loss functions; wherein, the multiple loss functions correspond to multiple prediction branches in the large language model; The large language model is jointly trained based on the weighted values of multiple loss values until the total loss function value converges to a preset value. When the convergence condition is met, the large language model is used as the highlight fragment detection model.
9. A video processing method, characterized in that, The method includes: Obtain the original video; Based on a second prompt text, a highlight fragment detection model extracts second media materials of various media types from the original video. These second media materials are then encoded to obtain second multimodal features. The original video is then analyzed based on these second multimodal features to obtain target highlight information. The highlight fragment detection model is trained using the training method described in any one of claims 1-8. The second prompt text is a task description text used to guide the highlight fragment detection model in performing highlight fragment detection based on the second media materials. Based on the target highlight information, an editing operation is performed on the original video to generate a target video corresponding to the original video.
10. The method according to claim 9, characterized in that, The method of extracting secondary media materials of various media types from the original video based on the secondary cue text using the highlight fragment detection model includes: Based on the second prompt text, the original video is extracted using a highlight fragment detection model; wherein, the extraction operation includes: dividing the original video into multiple second video segments according to a preset time length; and extracting the second dialogue text, second video transition points, and second audio segments corresponding to each second video segment; Each of the second video segments and its corresponding second dialogue text, second video transition points, and second audio segments are identified as the second media material.
11. The method according to claim 9, characterized in that, The target highlight information includes at least: target start and end time, target highlight type, target character information, and target title information; the step of performing editing operations on the original video based on the target highlight information to generate a target video corresponding to the original video includes: Edit multiple original highlight clips corresponding to the start and end times of the target from the original video; Based on the target highlight type, the target character information, and the target title information, video tags and / or visual materials are added to the original highlight clip to obtain the target highlight clip; The target highlight segment is processed and sorted to obtain the target video.
12. A training device for a specular fragment detection model, characterized in that, The device includes: A sample acquisition module is used to acquire video samples; wherein the video samples are marked with highlight reference information; wherein the highlight reference information is used to represent information of highlight segments as a reference standard; The highlight prediction module is used to extract first media materials of multiple media types from the video sample based on the first prompt text using a large language model, encode the first media materials to obtain first multimodal features, and perform highlight analysis on the video sample based on the first multimodal features to obtain highlight prediction information. The model training module is used to iteratively train the large language model based on the difference between the specular reference information and the specular prediction information; wherein, after completing the iterative training, the large language model is used as a specular fragment detection model.
13. A video processing apparatus, characterized in that, The device includes: The video acquisition module is used to acquire the original video. A highlight detection module is used to extract second media materials of multiple media types from the original video based on a second prompt text using a highlight fragment detection model, encode the second media materials to obtain second multimodal features, and perform highlight analysis on the original video based on the second multimodal features to obtain target highlight information; wherein, the highlight fragment detection model is trained using the training method of the highlight fragment detection model according to any one of claims 1-8; the second prompt text is a task description text used to guide the highlight fragment detection model to perform a highlight fragment detection task based on the second media materials; The video editing module is used to perform editing operations on the original video based on the target highlight information to generate a target video corresponding to the original video.
14. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-11.
15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method as described in any one of claims 1-11.
16. A storage medium, characterized in that, The storage medium is used to store the target video obtained based on any one of the video processing methods described in claims 8-10.
17. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 11.
