High-energy detection model training and method and device for detecting high-energy segments in video

Abstract

The present disclosure relates to a high-energy detection model training method and a method for detecting high-energy segments in a movie and television video, a device, equipment and medium. The training method comprises: obtaining a data set, the data set comprising a plurality of sample videos of multiple movie and television types and a video-level label of each sample video, each sample video being divided into a plurality of sample segments, and the video-level label being used to indicate whether a high-energy segment exists in the sample video; using the video-level label of each sample video in the data set to perform first-stage training on an initial detection model to obtain an intermediate detection model; using a segment-level label of each sample segment in each sample video generated by a large language model to perform second-stage training on the intermediate detection model to obtain a trained high-energy detection model, the segment-level label representing the high-energy degree of the sample segment. Thus, a high-energy detection model suitable for a movie and television scene can be obtained, the model can accurately locate high-energy segments in a movie and television video, and does not require a large amount of segment-level manual annotation.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to a method, apparatus, device and medium for training a high-energy detection model and detecting high-energy segments in film and television videos. Background Technology

[0002] With the rise of short videos, viewers' preferences have shifted from complete film and television content to short, emotionally intense clips, such as intense emotional conflicts, fight scenes, or plot twists. This trend has driven the growing demand for high-energy video clip detection: identifying the most engaging high-energy clips from the original long videos of films and television series to support subsequent content editing, recommendations, advertising, and derivative videos. High-energy clips typically refer to segments with strong emotions, fast-paced rhythms, and strong visual or auditory impact. Therefore, current research trends focus on using high-energy video detection models to automatically identify high-energy clips in long videos, helping creators quickly locate high-energy content and reducing the cognitive load before editing such clips. However, high-energy video detection for movies and television series remains challenging.

[0003] Existing high-energy detection models are typically designed for simple motion videos or log videos. Unlike sports or log videos where highlights follow structured patterns (such as goals or stunts), movies and TV shows have diverse genres, and the high-energy moments within them are varied. This significantly reduces the generalization ability and accuracy of existing detection models when dealing with such content. Furthermore, existing high-energy detection models generally rely on segment-level manual annotation of long videos (i.e., meticulous annotation of the segments containing high-energy moments). However, in diverse film and television scenarios, segment-level manual annotation is extremely costly. Moreover, the subjective and diverse nature of high-energy moments in movies and TV shows makes segment-level manual annotation prone to introducing the annotator's subjective bias, making it difficult to obtain large-scale, consistent annotation data, thus hindering model training and generalization. Summary of the Invention

[0004] In view of this, this disclosure proposes a high-energy detection model training method, device, equipment and medium for detecting high-energy segments in film and television videos. It can obtain a high-energy detection model adapted to film and television scenes, accurately locate high-energy segments in film and television videos, and does not require a large amount of segment-level manual annotation, thus avoiding the introduction of subjective bias by the annotators.

[0005] According to one aspect of this disclosure, a method for training a high-energy detection model for detecting high-energy segments in film and television videos is provided, comprising: acquiring a dataset, the dataset including multiple sample videos of various film and television types and video-level tags for each sample video, each sample video being segmented into multiple sample segments, the video-level tags being used to indicate whether a high-energy segment exists in the sample video; using the video-level tags of each sample video in the dataset to perform a first-stage training on an initial detection model to obtain an intermediate detection model; and using the segment-level tags of each sample segment in each sample video generated by a large language model to perform a second-stage training on the intermediate detection model to obtain a trained high-energy detection model, wherein the segment-level tags characterize the high-energy level of the sample segment.

[0006] In one possible implementation, the step of using the video-level labels of each sample video in the dataset to train the initial detection model in the first stage to obtain the intermediate detection model includes: for each sample video, using a pre-trained encoder to extract the visual and audio features of each sample segment in each sample video; using the initial detection model to generate a high-energy score for each sample segment in each sample video based on the visual and audio features of each sample segment in each sample video, wherein the high-energy score represents the probability that the sample segment is high-energy; for any sample video, selecting the K sample segments with the highest high-energy scores in the sample video, and using the average of the high-energy scores of the K sample segments as the video-level prediction result of the sample video; and using the loss between the video-level labels of each sample video and the video-level prediction results of each sample video to optimize the model parameters of the initial detection model to obtain the intermediate detection model.

[0007] In one possible implementation, the segment-level labels generated by the large language model for each sample video are used to perform a second-stage training on the intermediate detection model to obtain a trained high-energy detection model. This includes: in the nth round of training, using the large language model based on the content description of each sample segment in each sample video and the (n-1)th round high-energy mode, generating the nth round segment-level label corresponding to each sample segment in each sample video. The high-energy mode includes high-energy feature descriptions under different film and television types. The content description is generated using the multimodal large language model based on sample segments, where 1 ≤ n ≤ N. The 0th round high-energy mode is summarized as the initial high-energy mode, and N is the total number of training rounds. Based on the nth round segment-level label corresponding to each sample segment in each sample video, the detection model of the (n-1)th round is trained in multiple batches to obtain the nth round detection model. The 0th round detection model is the intermediate detection model. Using the large language model, based on the high-energy score of each sample segment in each sample video, the content description of each sample segment and the corresponding film and television type output in the multiple batches of iterative training, the nth round high-energy mode is generated. Among them, the Nth round detection model obtained after the Nth round of training is the trained high-energy detection model.

[0008] In one possible implementation, the step of performing multi-batch iterative training on the (n-1)th round detection model based on the nth round segment-level labels corresponding to each sample segment in each sample video to obtain the nth round detection model includes: in the m-th batch training of the nth round, using the (m-1)th generation detection model based on the visual and audio features of each sample segment in the m-th batch of videos, generating prediction results for the m-th batch of videos, wherein the prediction results include the high-energy score of each sample segment in the m-th batch of videos, 1≤m≤M; the M batches of videos are samples from the dataset. The video is segmented, and the 0th generation detection model is the (n-1)th round detection model. Using the prediction results of the m-th batch of videos, the (m-1)th generation segment-level labels, and the nth round segment-level labels, the target loss is determined, where the 0th generation segment-level labels are the nth round segment-level labels. The target loss is used to optimize the model parameters of the (m-1)th generation detection model and the (m-1)th generation segment-level labels to obtain the m-th generation detection model and the m-th generation segment-level labels. The M-th generation detection model obtained from the M-th batch of training in the nth round is the nth round detection model.

[0009] In one possible implementation, determining the target loss using the prediction results of the m-th batch of videos, the (m-1)-th generation segment-level labels, and the n-th round segment-level labels includes: determining a segment classification loss based on the prediction results of the m-th batch of videos and the (m-1)-th generation segment-level labels, wherein the segment classification loss is used to make the prediction results output by the detection model as close as possible to the segment-level labels; calculating a compatibility loss based on the (m-1)-th generation segment-level labels and the n-th round segment-level labels, wherein the compatibility loss is used to ensure that the optimized segment-level labels do not deviate excessively from the n-th round segment-level labels; determining a sharpness loss based on the prediction results of the m-th batch of videos, wherein the sharpness loss is used to encourage the high-energy scores output by the detection model to approach a maximum or minimum extreme value, wherein the high-energy scores output by the detection model are between the maximum and minimum extreme values; and determining the target loss based on the segment classification loss, the compatibility loss, and the sharpness loss.

[0010] In one possible implementation, the step of generating the nth round high-energy mode using the large language model based on the high-energy score of each sample segment in each sample video output during the multi-batch iterative training of the n-1th round detection model, the content description of each sample segment, and the corresponding film and television type includes: selecting multiple target sample segments with high-energy scores higher than a preset threshold based on the high-energy score of each sample segment in each sample video output during the multi-batch iterative training of the n-1th round detection model; and generating the nth round high-energy mode using the large language model based on the content description of the multiple target sample segments and the corresponding film and television type.

[0011] In one possible implementation, acquiring the dataset includes: acquiring multiple trending short videos, wherein the trending short videos are those with a popularity higher than a preset index value; filtering out trending short videos that do not involve film and television content based on the video titles of each trending short video to obtain multiple valid short videos, and segmenting each valid short video into multiple valid segments; using video fingerprinting technology to determine the segment of the original film and television video to which each valid segment in each valid short video belongs, and counting the number of times the same segment in the same original film and television video is used in the short video; based on the number of times the same segment in the same original film and television video is used in the short video, selecting sample videos from the original film and television videos and obtaining video-level tags for each sample video.

[0012] According to another aspect of this disclosure, a method for detecting high-energy segments in film and television videos is provided, comprising: acquiring a film and television video to be detected and dividing the film and television video into multiple video segments; extracting visual and audio features of each video segment in the film and television video using a pre-trained encoder; outputting a high-energy score for each video segment in the film and television video based on the visual and audio features of each video segment in the film and television video using a high-energy detection model; wherein the high-energy detection model is trained using the training method described above; and selecting video segments with high-energy scores higher than a specified threshold as detected high-energy segments.

[0013] According to another aspect of this disclosure, a training apparatus for a high-energy detection model for detecting high-energy segments in film and television videos is provided, comprising: an acquisition module for acquiring a dataset, the dataset including multiple sample videos of various film and television types and video-level tags for each sample video, each sample video being segmented into multiple sample segments, the video-level tags indicating whether a high-energy segment exists in the sample video; a first-stage training module for performing a first-stage training on an initial detection model using the video-level tags of each sample video in the dataset to obtain an intermediate detection model; and a second-stage training module for performing a second-stage training on the intermediate detection model using segment-level tags of each sample segment in each sample video generated by a large language model to obtain a trained high-energy detection model, the segment-level tags representing the high-energy level of the sample segment.

[0014] According to another aspect of this disclosure, a device for detecting high-energy segments in film and television videos is provided, comprising: a video acquisition module for acquiring a film and television video to be detected and segmenting the film and television video into multiple video segments; a feature extraction module for extracting visual and audio features of each video segment in the film and television video using a pre-trained encoder; a high-energy detection module for outputting a high-energy score for each video segment in the film and television video based on the visual and audio features of each video segment in the film and television video using a high-energy detection model; wherein the high-energy detection model is trained using the aforementioned training method; and a selection module for selecting video segments with high-energy scores higher than a specified threshold as detected high-energy segments.

[0015] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0016] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0018] According to various aspects of this disclosure, by first performing a weakly supervised pre-training of the initial detection model using video-level labels, an intermediate detection model is obtained. This is equivalent to initializing a model capable of detecting high-energy segments. Furthermore, the sample videos in the dataset contain various film and television genres, which can improve the model's generalization ability across genres. Then, a large language model is used as an automated segment-level labeler to provide fine-grained segment-level labels for model training. The intermediate detection model is then trained in the second stage to obtain a high-energy detection model that is deeply adapted to film and television scenes. This model can accurately locate high-energy segments in film and television content, has high recognition accuracy and generalization ability, and uses a large language model to generate segment-level labels, eliminating the need for manual annotation of which segments in the video are high-energy segments. This reduces the reliance on manual annotation, avoids the influence of the subjective bias of the annotators, and helps improve the detection accuracy of the trained high-energy detection model for high-energy segments.

[0019] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0021] Figure 1 A flowchart is shown illustrating a high-energy detection model training method for detecting high-energy segments in film and television videos according to an embodiment of the present disclosure.

[0022] Figure 2 A schematic diagram of the network structure of a lightweight detection model according to an embodiment of the present disclosure is shown.

[0023] Figure 3 A schematic diagram of a two-stage training framework according to an embodiment of the present disclosure is shown.

[0024] Figure 4 The diagram shows a block diagram of a high-energy detection model training apparatus for detecting high-energy segments in film and television videos according to an embodiment of the present disclosure.

[0025] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0026] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0027] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0028] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0029] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0030] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0031] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0032] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0033] As mentioned above, the rise of short videos has driven a growing demand for high-energy video clip detection, especially in the film and television industry. This often involves secondary creation, such as creating trailers and mashups. A high-quality secondary creation video not only needs to be understandable to viewers but also highly engaging. However, to achieve this, creators often need to invest a significant amount of time repeatedly watching the original long videos of films and television series, relying on personal experience to identify high-energy clips before editing—a highly inefficient process. Meanwhile, with the rise of Artificial Intelligence Generated Content (AIGC), the demand for AI-powered secondary creation videos is increasingly strong. Efficiently detecting high-energy clips in films and television series is key to assisting AI in improving the quality and efficiency of secondary video creation.

[0034] However, existing high-energy detection models are insufficient in film and television scenarios and cannot meet practical needs. Film and television high-energy recognition faces several core challenges: the diverse content of movies and TV series makes existing detection models unsuitable for the high-energy detection requirements of the film and television field; the diversity of high-energy plots in films and TV series makes it difficult to predefine high-energy content, leading to significant subjectivity in annotation; and fragment-level manual annotation of high-energy segments is both time-consuming and costly, easily affected by the subjective bias of the annotators, and makes it difficult to collect standardized manual annotation data on a large scale, thus restricting model training and generalization.

[0035] In view of this, this disclosure proposes a high-energy detection model training method based on a Large Language Model (LLM) without human collaboration. Specifically, it designs a two-stage training method of weakly supervised pre-training + LLM-guided self-iterative optimization, which can complete the training of the video high-energy detection model without manual definition / labeling. In simple terms, the first stage is to perform weakly supervised pre-training based on video-level labels to obtain a pre-trained intermediate detection model. In the second stage, a multimodal large language model (MLLM) and a large language model (LLM) are used as fragment-level labelers to provide fine-grained fragment-level labels for the model trained in the first stage, so as to iteratively train the intermediate detection model to obtain a high-energy detection model that is deeply adapted to the film and television scene. The high-energy detection model trained through the above two stages will accurately locate high-energy segments in film and television content, which is conducive to the accurate editing of high-energy segments in subsequent editing stages, effectively controlling the quality of the final product, significantly improving the creation efficiency of trailers, mashups and other derivative videos, enhancing the excitement of derivative videos and the user viewing experience, and generating exciting derivative videos for the recommendation system, effectively increasing content click-through rates and user consumption time.

[0036] The training and detection methods of this disclosure can be deployed on various terminal devices through software or hardware modifications. The terminal devices involved in this disclosure can refer to devices with wireless and / or wired connection functions. Wireless connection means that they can connect to other devices via wireless methods such as Wi-Fi and Bluetooth. The terminal devices involved in this disclosure can also communicate with other devices via wired connection. The terminal devices involved in this disclosure can be touchscreen, non-touchscreen, or screenless. Touchscreen devices can be controlled by clicking or swiping on the display screen using fingers or styluses. Non-touchscreen devices can connect to input devices such as mice, keyboards, and touch panels to control the terminal device. Screenless devices can be, for example, screenless Bluetooth speakers. For example, the terminal devices of this application can include, but are not limited to, user devices, mobile devices, user terminals, terminals, handheld devices, tablet computers, laptops, PDAs, and computing devices.

[0037] The training and detection methods of this disclosure can also be deployed on a server, which can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine or container, with wireless communication capabilities. These wireless communication capabilities can be configured in the server's chip (system) or other components. This can refer to a device with wireless connectivity, meaning it can connect to other servers or terminal devices via wireless connections such as Wi-Fi or Bluetooth. The server involved in this disclosure can also have wired communication capabilities. For example, the server in this disclosure can be located in the cloud, communicating with terminal devices, receiving datasets sent by the terminal devices, and using the training method deployed on the server to obtain a trained high-energy detection model; it can also receive video clips sent by the terminal devices and use the detection method deployed on the server to obtain the detected high-energy segments in the video clips.

[0038] Figure 1 A flowchart illustrating a high-energy detection model training method for detecting high-energy segments in film and television videos according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes steps S11 to S13.

[0039] In step S11, a dataset is obtained, which includes multiple sample videos of various film and television types and video-level tags for each sample video. Each sample video is divided into multiple sample segments, and the video-level tags are used to indicate whether there are high-energy segments in the sample video.

[0040] It is understandable that for a particular film or television drama, the more creators edit the same segment from that drama, the higher the quality of that segment. Similarly, the higher the popularity of derivative videos generated from that segment on social media platforms (such as short video platforms), the higher the quality of that segment. Therefore, in some embodiments, a dataset can be constructed by collecting short videos with a certain level of popularity related to the film or television content from social media platforms. Specifically, in some embodiments, obtaining the dataset may include:

[0041] Get multiple trending short videos; trending short videos are those whose popularity exceeds a preset indicator value.

[0042] Based on the video titles of various popular short videos, popular short videos that do not involve film and television content are filtered out to obtain multiple effective short videos, and each effective short video is divided into multiple effective segments;

[0043] Video fingerprinting technology is used to identify the original video clip to which each valid segment in each valid short video belongs, and the number of times the same segment in the same original video clip is used in the short video is counted.

[0044] Based on the number of times the same segment from the same original video is used in short videos, sample videos are selected from the original video and video-level tags are obtained for each sample video.

[0045] The popularity of short videos can be reflected in multiple aspects such as the number of views, likes, shares, comments, and favorites. It should be understood that those skilled in the art can customize and design the calculation rules for popularity indicators to determine the popularity of each short video based on the number of views, likes, shares, comments, and favorites. At the same time, they can customize and design preset indicator values ​​to identify short videos with popularity values ​​higher than the preset indicator values ​​as popular short videos. Thus, popular short videos can be obtained from social media platforms. This disclosure does not limit the method and source of obtaining popular short videos.

[0046] Understandably, short videos on the platform usually have video titles, which may include a summary of the video's content, related topics, and other information. Therefore, a large language model can be used to analyze the video titles of various trending short videos to perform an initial labeling of the trending short videos, that is, to label whether the trending short videos are related to film and television content. Then, trending short videos that do not involve film and television content can be filtered out to obtain multiple valid short videos. Considering that a short video may be generated by a mashup of film and television content, such as possibly coming from multiple films and TV series, or different segments from the same film and television series, each valid short video can be divided into multiple valid segments. For example, each valid short video can be divided into multiple non-overlapping segments of 2 seconds in length. This embodiment of the disclosure does not limit this.

[0047] In practical applications, those skilled in the art can use existing video fingerprinting technology (or a service system providing video fingerprinting technology) to extract and compare fingerprint features such as images and audio in valid segments. This allows them to locate the original video segment to which each valid segment belongs in each valid video from the film and television video library. In other words, it allows them to locate which segment of which film and television video the valid segment belongs to. Furthermore, it allows them to count the number of times the same segment from the same original film and television video is used (i.e., edited) in short videos. For example, valid short video A includes valid segments a1, a2, and a3, and valid short video B includes valid segments b1 and b2. If, through video fingerprinting, a1 is located to segment p1-1 of the original film and television video P1 (i.e., a1 is located to be a segment corresponding to a certain time period of the original film and television video P1; it should be understood that it is not necessary to segment the original film and television video, but only to locate which time period of the original film and television video the valid segment belongs to), and a2 is located to be a segment of the original film and television video P1, then... Segments p1-2 and a3 belong to segment p2-1 of the original video P2, b1 belongs to segment p1-1 of the original video P1, and b2 belongs to segment p2-1 of the original video P2. Therefore, segment p1-1 of the original video P1 is used twice in the short video (that is, both valid short video A and valid short video B use segment p1-1), segment p1-2 is used once, and segment p2-1 of the original video P2 is used twice. In this way, a voting and scoring mechanism can be used to use the number of times the segment in the original video is used as the voting score of the segment. The higher the score, the more likely the segment is to be a widely recognized high-energy scene. For example, a segment in the original video that is used more than 5 times can be considered to be a high-energy scene. Since a segment used more than 5 times in a short video is not necessarily a high-energy segment or a complete high-energy segment, it can be considered that there are high-energy segments in the original video containing segments used more than 5 times in the short video.

[0048] The original video footage can be a long movie or a long episode of a TV series. In some embodiments, original video footage in which the same segment is used in short videos more than a certain threshold can be used as a sample video containing high-energy segments. Alternatively, considering that movies are long and episodes of TV series are also long, a longer video segment (e.g., 5 minutes) containing segments used more than a certain threshold can be extracted from the original video footage as a sample video containing high-energy segments. For sample videos without high-energy segments, original video footage in which the number of uses does not exceed the threshold (or even not used in short videos at all) or long video segments (e.g., 5-minute long video segments) extracted from the original video footage that do not contain segments used more than the threshold can be selected as sample videos without high-energy segments. At the same time, a video-level tag for each sample video can be obtained. For example, the video-level tag can use 0 and 1 to identify whether there are high-energy segments in the sample video, and this disclosure does not limit this.

[0049] In this embodiment, a dataset is constructed by collecting trending short videos and combining them with a voting and scoring mechanism. This allows for the automated generation of sample data and its video-level tags, and is not limited to a single type of film or television content, enabling the dataset to cover sample videos from various film and television genres. Furthermore, it eliminates the need for pre-defining high-energy content and manual annotation, avoiding the subjective biases introduced by manual annotation and reducing the cost of dataset construction. In practical applications, after manual evaluation, the precision and recall rates of the video-level tags for sample videos in the dataset constructed based on trending short videos and the voting and scoring mechanism reached 0.8+, demonstrating that the dataset constructed in this embodiment has high annotation quality.

[0050] In some embodiments, the dataset can also be constructed by combining manual annotation with a voting scoring mechanism. For example, sample videos can be generated in advance based on various types of original film and television videos (such as directly using original film and television videos or extracting long video clips of 5 minutes). Multiple annotators manually annotate whether there are high-energy segments in the sample videos. Then, combined with a voting scoring mechanism (i.e., the more annotators believe that there are high-energy segments in the sample videos, the greater the probability that there are high-energy segments in the sample videos), video-level labels for the sample videos are generated. This eliminates the need for annotators to specifically annotate fine-grained high-energy segments, which can reduce the annotation workload to a certain extent. Furthermore, the voting scoring mechanism can reduce the subjective bias of annotators in identifying high-energy segments to a certain extent.

[0051] In practical applications, after obtaining the sample video, each sample video can be divided into multiple sample segments according to a preset unit duration (such as 2 seconds) to achieve subsequent fine-grained processing.

[0052] In step S12, the initial detection model is trained in the first stage using the video-level labels of each sample video in the dataset to obtain the intermediate detection model.

[0053] In practical applications, those skilled in the art can construct an initial detection model by referring to any existing deep learning model in the field (such as the Transformer model). The initial detection model can employ a lightweight network structure, possessing the ability to fuse visual and audio features for high-energy segment reasoning. Furthermore, a lightweight network structure is beneficial for ensuring efficient online reasoning of video and audio content. This disclosure does not limit the network structure of the detection model; exemplarily, Figure 2 This diagram illustrates a network structure of a lightweight detection model provided in an embodiment of the present disclosure, as shown below. Figure 2 As shown, the detection model includes: two uni-modal encoding layers consisting of Transformer layers, a cross-modal encoding module (composed of multiple multi-head attention layers, multiple fusion and normalization layers (Add&Norm), and a feedforward network layer), a fully connected layer (concat), a Local Attention Transformer layer, and a decoder (Detector). The two uni-modal encoding layers encode the input visual and audio features, respectively. The cross-modal encoding module interacts to enhance the encoded visual and audio features output by the two uni-modal encoding layers, thereby enhancing both visual and audio features. Following the implementation of the UMT framework proposed in the paper "UMT: unified multi-modal transformers for joint videomoment retrieval and highlight detection" by Y Liu et al., a bottleneck marker is introduced in the cross-modal encoding module as a cross-modal information bridge to capture and propagate visual or audio features, such as... Figure 2As shown, in the first multi-head attention layer of the cross-modal coding module, z is used as the query (q) in multi-head attention, and z' is used as the key (k) and value (v) in subsequent multi-head attention layers. z and z' are the input and output of the bottleneck marker, respectively. It should be understood that the specific implementation of the bottleneck marker is based on the relevant content in the aforementioned paper, and will not be elaborated here. The fully connected layer is used to fuse the visual and audio features after cross-modal interaction enhancement. Since the high-energy level of a segment may be affected by its context, in order to reduce interference from distant and irrelevant segments, the Transformer layer based on local attention is implemented with reference to the existing Swin transformer model. Local self-attention windows are introduced into the Transformer layer. Using local attention windows, the calculation of attention can be restricted to a local range rather than a global range, and contextual features with local temporal relationships between adjacent segments within the window can be extracted. The decoder is used to decode the contextual features output by the Transformer layer based on local attention to output the high-energy score of each segment in the video. The high-energy score represents the probability that the segment is high-energy, or the degree of high energy of the segment. The higher the high-energy score, the higher the energy of the segment.

[0054] It should be noted that the above Figure 2 The network structure shown is one possible implementation provided by the embodiments of this disclosure. In fact, under the guidance of the embodiments of this disclosure, those skilled in the art can customize the network structure of the detection model, and the embodiments of this disclosure do not limit this.

[0055] In practical applications, the Multiple Instance Learning (MIL) approach can be referenced to utilize the video-level labels of each sample video in the dataset to perform the first stage training of the initial detection model. That is, the sample videos can be considered as "bags," and the sample segments within them as "instances." Bags containing high-energy segments are positive bags, and bags not containing high-energy segments are negative bags. Using the known bags and their labels, the initial detection model is trained to initially learn the high-energy patterns (i.e., features of high-energy scenes) of segments from the video-level labels. In other words, the initial detection model is trained to learn what kinds of videos contain high-energy segments. Therefore, in some embodiments, in step S12 above, the initial detection model is trained using the video-level labels of each sample video in the dataset to obtain an intermediate detection model, including:

[0056] Step S121: For each sample video, use a pre-trained encoder to extract the visual and audio features of each sample segment in each sample video;

[0057] Step S122: Using the initial detection model, a high-energy score is generated for each sample segment in each sample video based on the visual and audio features of each sample segment in each sample video. The high-energy score represents the probability that the sample segment is high-energy.

[0058] Step S123: For any sample video, select the K sample segments with the highest high-energy scores in the sample video, and take the average high-energy scores of the K sample segments as the video-level prediction result of the sample video.

[0059] Step S124: Optimize the model parameters of the initial detection model by using the loss between the video-level labels of each sample video and the video-level prediction results of each sample video, and obtain the intermediate detection model.

[0060] In step S121, the pre-trained encoder can be a pre-trained encoder used to extract visual and audio features from the video (for example, a pre-trained visual encoder and an audio encoder can be used respectively), as long as it can extract visual and audio features, this embodiment of the disclosure does not limit this. It should be understood that extracting the visual and audio features of each sample segment in any sample video is actually extracting the visual and audio features of the entire sample video.

[0061] In step S122, for each sample video, the visual and audio features of each sample segment in the sample video can be input into the initial detection model to obtain the high-energy score of each sample segment in the sample video. Then, in step S123, the K sample segments with the highest high-energy scores in the sample video (i.e., the top-K high-energy score sample segments, where K is a user-defined positive integer) can be selected. This is equivalent to selecting the segment with the highest confidence in each packet (positive or negative packet), or the segment with the most information content, and the average high-energy score of the top-K high-energy score sample segments (i.e., the average score) is calculated as the video-level prediction result of the sample video. It should be understood that through the above steps S121 to S123, the video-level prediction result of each sample video can be obtained, which can reflect the probability of the presence of high-energy segments in the sample video predicted by the model.

[0062] In step S124, for example, the packet classification loss (e.g., binary cross-entropy loss) between the video-level labels of each sample video and the video-level prediction results of each sample video can be calculated. The packet classification loss is used to optimize the model parameters of the initial detection model to obtain an intermediate detection model. This embodiment of the present disclosure does not limit the specific process of optimizing model parameters using the packet classification loss. It should be understood that steps S121 to S124 can be iteratively executed multiple times until preset conditions are met (such as reaching the required number of execution rounds, loss convergence, etc.) to obtain the trained intermediate detection model. This embodiment of the present disclosure does not limit this.

[0063] It should be understood that the first stage of training enables the intermediate detection model to initially possess the ability to detect high-energy segments from videos. However, since the training process uses coarse-grained video-level labels, the intermediate detection model's recognition accuracy for high-energy segments is not high. In order to further improve the recognition accuracy of the intermediate detection model, the second stage of training is conducted using fine-grained segment-level labels generated by the large language model in subsequent step S13. This allows the trained high-energy detection model to have higher recognition accuracy.

[0064] In step S13, the intermediate detection model is trained in the second stage using the segment-level labels of each sample segment in each sample video generated by the large language model, so as to obtain a trained high-energy detection model. The segment-level labels represent the high-energy level of the sample segments.

[0065] In practical applications, the logical reasoning capabilities of large language models can be leveraged to generate fine-grained but noisy fragment-level labels. These fragment-level labels are then used to perform Noisy Label Cleaning (NLC) on intermediate detection models, transforming the fragment-level labels generated by the large language model into learnable probability distributions (i.e., converting fragment-level labels into trainable parameters equivalent to model parameters). During training, model parameters and labels are simultaneously optimized, with adaptive label correction and model parameter optimization. In each training round, high-confidence fragments (i.e., sample fragments with high-energy scores) output by the detection model can be used by the large language model to inductively extract high-energy patterns (i.e., summarizing high-energy feature descriptions for different film and television genres, such as fight scene features in martial arts dramas and emotional outburst scene features in romance dramas). In this embodiment of the disclosure, the high-energy feature description under different film and television genres can be understood as the features of high-energy plots under different film and television genres. For example, high-energy plots in martial arts dramas are usually fighting plots, while high-energy plots in romance dramas are usually emotional outburst plots. The high-energy feature description of martial arts dramas can include features describing fighting plots, and the high-energy feature description of romance dramas can include features describing emotional outburst plots. This embodiment of the disclosure does not limit this. Using a large language model to extract high-energy patterns based on high-confidence fragments can be understood as summarizing what plot features are high-energy under different film and television genres. These extracted high-energy patterns can be used as prompt words input into the large language model to guide the large language model to generate cleaner fragment-level labels for subsequent training rounds, improving the accuracy of the fragment-level labels output by the large language model in the next round. After multiple rounds of the above training, the high-energy recognition accuracy of the high-energy detection model can be effectively improved. Therefore, specifically, step S13 above, using the fragment-level labels of each sample video generated by the large language model to perform the second stage of training on the intermediate detection model to obtain the trained high-energy detection model, can include:

[0066] Step S131: In the nth round of training, the large language model is used to generate the nth round segment-level label corresponding to each sample segment in each sample video based on the content description of each sample segment in each sample video and the high-energy mode of the (n-1)th round. The high-energy mode includes high-energy feature descriptions under different film and television types. The content description is generated based on the sample segments using the multimodal large language model, 1≤n≤N, the high-energy mode of the 0th round is summarized as the initial high-energy mode, and N is the total number of training rounds.

[0067] Step S132: Based on the nth round segment-level label corresponding to each sample segment in each sample video, perform multiple batches of iterative training on the (n-1)th round detection model to obtain the nth round detection model. The 0th round detection model is the intermediate detection model.

[0068] Step S133: Using the large language model, based on the high-energy score of each sample segment in each sample video output during multi-batch iterative training, the content description of each sample segment and the corresponding film and television type, generate the nth round of high-energy mode;

[0069] Among them, the detection model obtained after the Nth round of training is the trained high-energy detection model.

[0070] In practical applications, the second-stage training can be performed for N epochs. In each epoch, the nth-epoch fragment-level labels used in the current epoch are generated first through step S131, which is to initialize the fragment-level labels for each epoch. Then, based on the fragment-level labels initialized in the current epoch, step S132 performs multi-batch iterative training on the detection model of the (n-1)th epoch after the previous epoch. Since the fragment-level labels output by the large language model are noisy, step S132 can be multi-batch iterative training to clean and learn noisy labels for the (n-1)th epoch detection model. During multi-batch iterative training, noisy labels are corrected and model parameters are optimized simultaneously to improve the model's generalization ability. Step S133 is to perform high-energy pattern induction based on the output of the detection model in the multi-batch iterative training of the nth epoch to improve the fragment-level labels generated by the large language model in the next epoch. This constitutes a self-optimizing loop that does not require manual annotation: the large language model generates labels, noisy labels are cleaned and learned, and high-energy patterns are summarized from the output of the detection model, which effectively improves the quality of fragment-level labels generated by the large language model in the next round of training.

[0071] In step S131, for each sample segment of each sample video, the content description of each sample segment of each sample video can be pre-input into the multimodal large language model (MLLM) to generate a content description of each sample segment of each sample video. The content description expresses the content of the sample segment, such as "a character is crying". In each round of training, the content description of each sample segment of each sample video and the high-energy mode generated in the previous round (n-1) can be input into the large language model, allowing the large language model to perform reasoning analysis using the CoT (Cooperative Thought) method to obtain the segment-level label generated by the large language model for each sample segment. For example, the large language model can first infer the film / TV genre corresponding to the sample segment based on the content description of the sample segment, and then evaluate the degree to which the content description of the sample segment conforms to the high-energy feature description of the corresponding video genre based on the film / TV genre, content description, and high-energy feature descriptions under different film / TV genres in the (n-1)th round of high-energy mode. Based on the evaluation result, the sample segment is scored, which is equivalent to outputting a high-energy score as a segment-level label for each sample segment. For example, the large language model can output a high-energy score with a granularity of 0.1, where the closer the score is to 1, the higher the degree of high energy.

[0072] In some embodiments, when constructing the dataset, the film and television type of each sample video can be known and labeled in advance by locating the original film and television videos (that is, the film and television type of each sample segment of each sample video is labeled). When generating segment-level labels using the large language model in step S131 above, the pre-labeled film and television types can also be input into the large language model at the same time. In this way, the thinking chain used by the large language model can directly use the film and television type, content description and high-energy mode corresponding to the input sample segment for evaluation and analysis and then output segment-level labels. This disclosure embodiment does not limit this.

[0073] In practical applications, the initial high-energy mode can be a pre-set initial high-energy mode. It should be understood that those skilled in the art can set the initial high-energy mode based on experience. This initial high-energy mode can be used in the first round of training, and the high-energy mode summarized by the large language model can be used in subsequent rounds of training. In the embodiments of this disclosure, by inputting the high-energy mode into the large language model, the large language model can be guided to generate higher-quality fragment-level labels.

[0074] In some embodiments, after generating the nth-round segment-level label corresponding to each sample segment in each sample video based on the content description of each sample segment in each sample video and the (n-1)th round of high-energy pattern induction using the large language model, the high-confidence segments (i.e., sample segments with high high-energy scores, such as sample segments above a certain threshold, or top-K sample segments) output by the intermediate detection model in the first training stage can be used to preliminarily optimize and enhance the segment-level labels generated by the large language model. For example, the deviation between the high-confidence segments output in the first training stage and the segment-level labels of the corresponding sample segments generated by the large language model can be determined first. If the deviation exceeds a certain threshold, the large language model can regenerate the segment-level labels or directly replace them with the high-energy scores of the high-confidence segments output in the first training stage. Then, the optimized nth-round segment-level labels are used to perform the noisy label cleaning learning in the subsequent step S132. In this way, the prediction results of the intermediate detection model with a certain degree of credibility can be used to reduce the noise level of the segment-level labels output by the large language model to a certain extent in advance, which is beneficial to improving the model training efficiency in the subsequent step S132. Of course, it is also possible to directly use the fragment-level labels generated by the large language model to perform subsequent step S132 without enhancing or optimizing the fragment-level labels generated by the large language model. This embodiment of the present disclosure does not limit this.

[0075] In practical applications, after generating the nth round fragment-level labels in step S131, the nth round fragment-level labels can be transformed into an updatable probability distribution, and this distribution can be registered as a trainable parameter, that is, transformed into a learnable variable parallel to the model parameters. This distribution is initialized as the nth round fragment-level labels, and in subsequent step S132, it can start from the noisy initial value (i.e., the nth round fragment-level labels) and continuously self-correct during multiple batches of iterative training.

[0076] Based on the noisy label cleaning learning method, in step S132, the detection model of the (n-1)th round is iteratively trained multiple times based on the segment-level label of the nth round corresponding to each sample segment in each sample video to obtain the nth round detection model, which may include:

[0077] Step S1321: In the m-th batch of training in the n-th round, the (m-1)-th generation detection model is used to generate the prediction result of the m-th batch of videos based on the visual and audio features of each sample segment in the m-th batch of videos. The prediction result includes the high-energy score of each sample segment in the m-th batch of videos, 1≤m≤M; the M batches of videos are obtained by dividing the sample videos in the dataset, and the 0th generation detection model is the (n-1)-th round detection model;

[0078] Step S1322: Using the prediction results of the m-th batch of videos, the (m-1)-th generation segment-level labels, and the n-th round segment-level labels, determine the target loss, where the 0th generation segment-level labels are the n-th round segment-level labels;

[0079] Step S1323: Optimize the model parameters of the (m-1)th generation detection model and the (m-1)th generation fragment-level labels using the target loss to obtain the m-th generation detection model and the m-th generation fragment-level labels;

[0080] Among them, the Mth generation detection model obtained from the Mth batch of training in the nth round is the nth round detection model.

[0081] As described above, visual and audio features of each sample segment in each sample video can be extracted using a pre-trained encoder, and this embodiment of the present disclosure does not limit this. In step S1321, the visual and audio features of each sample segment in the m-th batch of videos can be input into the (m-1)-th generation detection model to obtain the high-energy score of each sample segment in the m-th batch of videos. It should be understood that those skilled in the art can customize the batch size according to actual conditions, dividing the sample videos in the dataset into multiple batches according to this batch size. A batch of videos includes a portion of the sample videos in the dataset. It should be understood that each round of training yields the detection model for that round (e.g., the n-th round detection model), and within each round of training, M-generation training is performed on M batches of sample videos. The detection model obtained by training using each batch of sample videos is the detection model for that batch (e.g., the m-th generation detection model). When m=1, the 0th generation detection model trained in the 1st batch of training in the nth round (i.e., the initial detection model trained in the nth round) is the (n-1)th round detection model obtained after training in the (n-1)th round. In other words, the initial training object of the current round is the model obtained after training in the previous round. When m is greater than 1, that is, the detection model trained in each batch from the 2nd to the Mth batches of training in the nth round is the detection model obtained after training in the previous batch.

[0082] It should be understood that the nth-round segment-level label generated through step S131 above is used as the initial segment-level label for multi-batch iterative training of the (n-1)th-round detection model. In step S1322, when m=1 (i.e., in the first batch of training), the 0th-generation segment-level label is the nth-round segment-level label; when m>1 (i.e., in the second to Mth batches of training), the (m-1)th-generation segment-level label is the segment-level label optimized using the target loss in the (m-1)th batch of training. In some embodiments, step S1322 above, using the prediction results of the mth batch of video, the (m-1)th-generation segment-level label, and the nth-round segment-level label to determine the target loss, may include:

[0083] Step S13221: Based on the prediction results of the m-th batch of videos and the segment-level labels of the (m-1)th generation, determine the segment classification loss. The segment classification loss is used to make the prediction results output by the detection model as close as possible to the segment-level labels.

[0084] Step S13222: Calculate the compatibility loss based on the (m-1)th generation fragment-level labels and the nth round fragment-level labels. The compatibility loss is used to ensure that the optimized fragment-level labels do not deviate excessively from the nth round fragment-level labels.

[0085] Step S13223: Based on the prediction results of the m-th batch of videos, determine the sharpness loss, which is used to encourage the high-energy score output by the detection model to approach the maximum or minimum extreme value, wherein the high-energy score output by the detection model is between the maximum and minimum extreme values.

[0086] Step S13224: Determine the target loss based on the segment classification loss, compatibility loss, and sharpness loss.

[0087] It should be understood that due to the noise in the segment-level labels output by the large model, directly using standard loss (such as cross-entropy) may lead to model overfitting to noise. Therefore, in step S13221, the label correction strategy in the NLC field can be referenced, and the KL divergence function can be used to calculate the packet classification loss. That is, the KL divergence can be used to measure the difference between the segment-level labels of the (m-1)th generation and the prediction results of the m-th batch of videos output by the detection model, thus obtaining the segment classification loss. For example, the segment classification loss... It can be calculated using formulas (1-1) and (1-2):

[0088] (1-1)

[0089] (1-2)

[0090] in, Represents the KL divergence function. =[ , ] represents the visual features of the i-th sample segment of the input. and audio features , This represents the detection model of generation m-1 in terms of model parameters. The high-energy score of the i-th sample segment generated below, where T represents the total number of sample segments in a batch of videos. Let represent the fragment-level label of the i-th sample fragment in the (m-1)-th generation fragment-level label, and p represent the number of categories (e.g., high-energy or non-high-energy). It should be understood that the fragment classification loss described above can make the prediction results output by the detection model as close as possible to the fragment-level labels, that is, make the distribution of the prediction results output by the detection model as close as possible to the distribution of the (m-1)-th generation fragment-level labels, so as to shorten the difference between the model output and the fragment-level labels.

[0091] In step S13222, it is understood that although the fragment-level labels generated by the large language model are noisy, most of the labels are accurate. If the fragment-level labels are arbitrarily and freely updated during batch training, the updated fragment-level labels may completely discard the initial fragment-level label information, leading to training out of control. Therefore, compatibility loss can be calculated to prevent the batch-updated fragment-level labels from deviating excessively from the initial fragment-level labels (i.e., the nth round fragment-level labels). For example, the compatibility loss... It can be calculated using formula (2):

[0092] (2)

[0093] in, This represents the fragment-level label in the nth round (i.e., the initial fragment-level label generated by the second largest language model in the nth round). This represents the fragment-level label of the i-th sample fragment in the n-th round of fragment-level labeling.

[0094] In step S13223, to prevent premature model convergence (i.e., both model output and label distribution stabilize at suboptimal intermediate values), a sharpness loss can be introduced to encourage the output to approach the maximum extremum (e.g., 1) and minimum extremum (e.g., 0). 1 represents a segment that is definitely a high-energy segment, and 0 represents a segment that is definitely not a high-energy segment. The high-energy score of the detection model output can be set between the maximum and minimum extrema; the closer to the maximum extremum, the higher the segment is considered; the closer to the minimum extremum, the lower the segment is considered. For example, the sharpness loss... It can be calculated using formula (3):

[0095] (3)

[0096] In step S13224, the target loss can be obtained, for example, by weighting and summing the segment classification loss, compatibility loss, and sharpness loss. That is, the target loss can be expressed as: , and It is a hyperparameter that controls the weights of compatibility loss and sharpness loss.

[0097] In step S1323, optimization algorithms such as backpropagation and gradient descent (or an optimizer such as the Adam optimizer) can be used to optimize the model parameters of the (m-1)th generation detection model and the (m-1)th generation fragment-level labels based on the target loss, thus obtaining the m-th generation detection model and the m-th generation fragment-level labels. This embodiment does not limit the above optimization process; for example, the target loss can be calculated separately for the model parameters. The model parameters and fragment-level labels are updated based on the gradients of the fragment-level labels. The model parameter updates can use a lower learning rate to gradually converge, while the fragment-level label updates can use a higher learning rate to accelerate the denoising process. This disclosure does not limit the implementation of these embodiments.

[0098] It should be understood that by iteratively executing steps S1321 to S1323 until training of M batches is completed, the Mth generation detection model after training of the Mth batch can be obtained. Therefore, the Mth generation detection model obtained by training the Mth batch in the nth round can be used as the nth round detection model after training in the nth round.

[0099] It should be understood that during the multi-batch iterative training of the detection model in the (n-1)th round, each generation of the detection model will output the high-energy score of each sample video in each batch of videos. That is, the high-energy score of each sample segment in each sample video output during the multi-batch iterative training of the detection model in the nth round can be obtained. Then, step S133 can be executed to generate the high-energy mode of the nth round by using the high-energy score of each sample segment in each sample video output during the multi-batch iterative training in the nth round of training, the content description of each sample segment and the corresponding film and television type.

[0100] In step S133, based on the high-energy scores of each sample segment in each sample video output during the multi-batch iterative training in the nth round of training, high-confidence segments (such as sample segments with high-energy scores higher than a certain threshold) can be extracted. These high-confidence segments can be considered as segments with a high probability of containing high-energy content. Then, based on the content descriptions of these high-confidence segments and the corresponding video types, the large language model can summarize typical high-energy patterns under different film and television types. Thus, in some embodiments, the above-mentioned use of the large language model to generate the nth round high-energy pattern based on the high-energy scores of each sample segment in each sample video output during the multi-batch iterative training of the n-1th round of detection model, the content descriptions of each sample segment, and the corresponding film and television types includes:

[0101] Step S1331: Based on the high-energy score of each sample segment in each sample video output during the multi-batch iterative training of the detection model in the (n-1)th round, select multiple target sample segments with high-energy scores higher than a preset threshold.

[0102] Step S1332: Using a large language model, generate the nth round of high-energy mode based on the content descriptions of multiple target sample segments and their corresponding film and television types.

[0103] In step S1331, the preset threshold can be customized according to the actual situation, and this embodiment of the present disclosure does not limit this.

[0104] In step S1332, for example, a high-quality high-energy pattern can be generated by instructing the large language model to perform three steps. For example, the first step instructs the large language model to analyze the content description of each target sample fragment and its corresponding film and television genre to generate analytical reasons why each target sample fragment might be a high-energy fragment. For example, if a character in a target sample fragment is crying, this target sample fragment belongs to the romance drama genre, so this sample fragment is likely to be high-energy. Alternatively, it is possible that a character in a target sample fragment is crying, this target sample fragment belongs to the martial arts drama genre, so this sample fragment may not be high-energy. The second step can aggregate and summarize the analytical reasons for the target sample fragments belonging to each film and television genre to obtain the high-energy feature descriptions corresponding to each film and television genre. Here, semantically similar high-energy feature descriptions can be merged, and low-frequency or redundant high-energy feature descriptions can be removed to form representative high-energy feature descriptions for each film and television genre. The third step allows the large language model to correct the high-energy feature descriptions of each film and television genre output in the second step, that is, to remove high-energy feature descriptions with common sense errors or logical errors, to obtain the nth round of high-energy patterns. Of course, the large language model can also be directly instructed to generate a high-energy mode based on the content description of multiple target sample segments and the corresponding film and television types. This disclosure does not limit this approach.

[0105] In this embodiment of the disclosure, the high-energy pattern induction based on the high-confidence fragments generated in each training round by the large language model can gradually improve the quality of the high-energy patterns generated by the large language model. The high-energy patterns are then injected into the fragment-level label generation in the next round, which can guide the large language model to generate high-quality fragment-level labels with less noise, thereby achieving iterative label refinement.

[0106] It should be understood that by executing steps S131 to S133 repeatedly until N rounds of training are completed, the Nth round of detection model after the Nth round of training can be obtained. Therefore, the Nth round of detection model obtained after the Nth round of training can be used as the trained high-energy detection model.

[0107] According to the training method of this disclosure, an intermediate detection model is obtained by first performing a weakly supervised pre-training on an initial detection model using video-level labels. This is equivalent to initializing a model capable of detecting high-energy segments. Since the sample videos in the dataset have multiple film and television types, the model's generalization ability across types can be improved. Then, a large language model is used as an automated segment-level labeler to provide fine-grained segment-level labels for model training. The intermediate detection model is then trained in the second stage to obtain a high-energy detection model that is deeply adapted to film and television scenes. This model can accurately locate high-energy segments in film and television content, has high recognition accuracy and generalization ability, and uses a large language model to generate segment-level labels, eliminating the need for manual annotation of which segments in the video are high-energy segments. This reduces the reliance on manual annotation, avoids the influence of the subjective bias of the annotators, and helps improve the detection accuracy of the trained high-energy detection model for high-energy segments.

[0108] It should be noted that the large language model used in the above embodiments of this disclosure can be any known large language model in the art. The large language model can be deployed locally, or it can be directly used on any online platform that provides large language model inference services. During the execution of the training method, the large language model can be invoked to complete the steps that the large language model needs to execute in the training method by sending instruction requests (such as the prompt word Prompt instruction) to the large language model. This disclosure does not limit this.

[0109] Based on the training method provided in the above embodiments of this disclosure. Figure 3 The diagram illustrates a two-stage training framework provided in one embodiment of this disclosure, as shown below. Figure 3As shown, the training framework includes: weakly supervised training in stage 1 and iterative collaborative training in stage 2. In stage 1, multiple sample videos generate visual and audio features through pre-trained encoders (including visual encoders and audio encoders), which are then input into the initial detection model to obtain the predicted label for each sample segment in each sample video (i.e., the high-energy score output by the detection model). The top-K sample segments with high-energy scores are selected, and the MIL loss (such as the bag classification loss mentioned above) is calculated in combination with the video-level labels to optimize the initial detection model. In Phase 2, the LLM-guided label generation module generates fragment-level labels based on high-energy patterns. These fragment-level labels are then converted into label distributions to participate in the calculation of NLC loss (i.e., the aforementioned target loss). Simultaneously, the detection model generates predicted labels (i.e., high-energy scores) for each sample fragment, calculates the NLC loss, and uses the NLC loss to optimize the detection model and fragment-level labels. Additionally, the high-energy pattern induction module generates high-energy patterns based on high-confidence fragments. In the large language model-guided label generation module, MLLM generates content descriptions for each sample fragment. LLM, based on each sample fragment and high-energy pattern, uses the COT (Consciousness-Oriented Trace) to generate fragment-level labels. The predicted labels generated in Phase 1 can also be used to optimize and enhance the fragment-level labels output by LLM. In the high-energy pattern induction module, the large language model uses metadata (including sample fragment content descriptions and film / television genres) of high-confidence fragments to perform pattern generation (i.e., generating analytical reasons why each target sample fragment might be a high-energy fragment), pattern aggregation (i.e., generating high-energy feature descriptions corresponding to each film / television genre), and pattern correction (correcting the high-energy feature descriptions for each film / television genre) to obtain high-energy patterns.

[0110] The training method according to the embodiments of this disclosure enables the trained high-energy detection model to have generalized high-energy recognition capabilities. It abandons the single detector design that relies on fixed high-energy patterns, introduces LLM for semantic understanding of film and television content, and adaptively captures high-energy features under different themes and narrative styles, significantly improving cross-theme generalization capabilities and no longer being limited to specific motion or scene types. It requires no language query input, using the video content itself to drive high-energy detection, truly achieving general and open-scene high-energy recognition of film and television, adapting to a wider range of practical application needs. Furthermore, it eliminates reliance on manual annotation, replacing manual segment-level annotation with real-world community behavior data (i.e., trending short videos), combining weakly supervised pre-training with automatic segment-level label generation by LLM, reducing annotation costs and annotator bias, and supporting large-scale training of large-scale film and television content. Finally, to address the pseudo-label noise problem caused by LLM illusion, a noise-robust training strategy is introduced, and a self-improvement closed loop between LLM and the detection model is constructed. The high-confidence predictions of the detection model are continuously fed back to LLM, driving iterative improvement of segment-level label quality and effectively transforming noisy labels into reliable training signals.

[0111] The training scheme of this disclosure embodiment trains a high-energy detection model that is deeply adapted to film and television scenes without the need for manual annotation or language query, and achieves accurate positioning of diverse high-energy segments in movies and TV series. Compared with existing detection models, the high-energy detection model obtained by the training method of this disclosure embodiment has better high-energy recognition effect.

[0112] The training scheme of this disclosure proposes a novel dataset construction method, which innovatively utilizes community editing behavior to automatically construct video-level labels, eliminating the reliance on manual annotation; it also proposes an LLM-guided unmanned collaborative training framework, which mines potential high-energy data from user behavior on short video platforms, and then introduces LLM-guided semantic iterative refinement to break through the performance bottleneck.

[0113] After training the high-energy detection model, it can be used to detect high-energy segments in film and television videos. Therefore, this disclosure proposes a method for detecting high-energy segments in film and television videos, including:

[0114] The program acquires the video to be detected and segments it into multiple video clips.

[0115] Visual and audio features of each video segment in a movie or TV show are extracted using a pre-trained encoder.

[0116] The high-energy detection model is used to output the high-energy score of each video segment in the film and television video based on the visual and audio features of each video segment; the high-energy detection model is trained using the method described above.

[0117] Video segments with high energy scores above a specified threshold are selected as the detected high-energy segments.

[0118] It should be understood that this embodiment of the disclosure does not limit the duration, type, or content of the video to be detected. The video can be segmented into multiple video segments by referring to the unit duration (e.g., 2 seconds) of the sample video segmentation in the model training described above. The pre-trained encoder (e.g., a visual encoder and an audio encoder) can use the encoder from the model training described above; this embodiment of the disclosure does not limit this. Then, the visual and audio features of each video segment in the video can be input into the high-energy detection model, which outputs a high-energy score for each video segment. Video segments with high-energy scores higher than a specified threshold are then selected as detected high-energy segments. It should be understood that the selected video segments may contain one or more groups of consecutive video segments, each group of consecutive video segments constituting a high-energy segment. Alternatively, it may include one or more discontinuous video segments, in which case each discontinuous video segment can be considered a high-energy segment; this embodiment of the disclosure does not limit this.

[0119] The detection method according to the embodiments of this disclosure can achieve accurate detection of high-energy segments in film and television videos using a high-energy detection model.

[0120] Figure 4 This diagram illustrates a block diagram of a high-energy detection model training apparatus for detecting high-energy segments in film and television videos according to an embodiment of the present disclosure. Figure 4 As shown, the device includes:

[0121] The acquisition module 401 is used to acquire a dataset, which includes multiple sample videos of various film and television types and video-level tags for each sample video. Each sample video is divided into multiple sample segments, and the video-level tags are used to indicate whether there are high-energy segments in the sample video.

[0122] The first-stage training module 402 is used to perform first-stage training on the initial detection model using the video-level labels of each sample video in the dataset to obtain an intermediate detection model.

[0123] The second-stage training module 403 is used to train the intermediate detection model in the second stage using the segment-level labels of each sample segment in each sample video generated by the large language model, so as to obtain a trained high-energy detection model. The segment-level labels represent the high-energy level of the sample segments.

[0124] In some embodiments, the step of using the video-level labels of each sample video in the dataset to perform a first-stage training on the initial detection model to obtain an intermediate detection model includes:

[0125] For each sample video, a pre-trained encoder is used to extract the visual and audio features of each sample segment in each sample video;

[0126] The initial detection model is used to generate a high-energy score for each sample segment in each sample video based on the visual and audio features of each sample segment. The high-energy score represents the probability that the sample segment is high-energy.

[0127] For any given sample video, select the K highest-scoring sample segments from that sample video, and use the average of the high-scoring scores of these K sample segments as the video-level prediction result for that sample video.

[0128] By utilizing the loss between the video-level labels of each sample video and the video-level prediction results of each sample video, the model parameters of the initial detection model are optimized to obtain an intermediate detection model.

[0129] In some embodiments, the segment-level labels of each sample video generated using the large language model are used to perform a second-stage training on the intermediate detection model to obtain a trained high-energy detection model, including:

[0130] In the nth round of training, the large language model is used to generate the nth round segment-level label corresponding to each sample segment in each sample video based on the content description of each sample segment in each sample video and the high-energy mode of the (n-1th round). The high-energy mode includes high-energy feature descriptions under different film and television types. The content description is generated based on the sample segments using the multimodal large language model. 1≤n≤N, the high-energy mode of the 0th round is summarized as the initial high-energy mode, and N is the total number of training rounds.

[0131] Based on the segment-level label of the nth round corresponding to each sample segment in each sample video, the detection model of the (n-1)th round is trained in multiple batches to obtain the detection model of the nth round, and the detection model of the 0th round is the intermediate detection model;

[0132] Using the large language model, based on the high-energy score of each sample segment in each sample video output during the multi-batch iterative training, the content description of each sample segment and the corresponding film and television type, the nth round of high-energy mode is generated;

[0133] Among them, the detection model obtained after the Nth round of training is the trained high-energy detection model.

[0134] In some embodiments, the step of performing multiple batches of iterative training on the (n-1)th round detection model based on the nth round segment-level labels corresponding to each sample segment in each sample video to obtain the nth round detection model includes:

[0135] In the m-th batch of training in the n-th round, the (m-1)-th generation detection model is used to generate the prediction results of the m-th batch of videos based on the visual and audio features of each sample segment in the m-th batch of videos. The prediction results include the high-energy score of each sample segment in the m-th batch of videos, 1≤m≤M; the M batches of videos are obtained by dividing the sample videos in the dataset, and the 0th generation detection model is the (n-1)-th round detection model;

[0136] The target loss is determined by using the prediction results of the m-th batch of videos, the (m-1)-th generation segment-level labels, and the n-th round segment-level labels, wherein the 0th generation segment-level labels are the n-th round segment-level labels;

[0137] The model parameters of the (m-1)th generation detection model and the (m-1)th generation fragment-level labels are optimized using the target loss to obtain the m-th generation detection model and the m-th generation fragment-level labels;

[0138] The Mth generation detection model obtained from the Mth batch of training in the nth round is the nth round detection model.

[0139] In some embodiments, determining the target loss using the prediction results of the m-th batch of videos, the (m-1)-th generation segment-level labels, and the n-th round segment-level labels includes:

[0140] Based on the prediction results of the m-th batch of videos and the segment-level labels of the (m-1)th generation, the segment classification loss is determined. The segment classification loss is used to make the prediction results output by the detection model as close as possible to the segment-level labels.

[0141] Based on the (m-1)th generation fragment-level labels and the nth round fragment-level labels, a compatibility loss is calculated. The compatibility loss is used to ensure that the optimized fragment-level labels do not deviate excessively from the nth round fragment-level labels.

[0142] Based on the prediction results of the m-th batch of videos, a sharpness loss is determined. The sharpness loss is used to encourage the high-energy score output by the detection model to approach the maximum or minimum extreme value, wherein the high-energy score output by the detection model is between the maximum and minimum extreme values.

[0143] The target loss is determined based on the segment classification loss, the compatibility loss, and the sharpness loss.

[0144] In some embodiments, the step of generating the nth round high-energy mode using the large language model based on the high-energy score of each sample segment in each sample video, the content description of each sample segment, and the corresponding film and television type output during the multi-batch iterative training of the n-1th round detection model includes:

[0145] Based on the high-energy scores of each sample segment in each sample video output during the multi-batch iterative training of the detection model in the (n-1)th round, multiple target sample segments with high-energy scores higher than a preset threshold are selected.

[0146] Using the large language model, based on the content descriptions of the multiple target sample segments and their corresponding film and television types, the nth round of high-energy mode is generated.

[0147] In some embodiments, obtaining the dataset includes:

[0148] Acquire multiple trending short videos, wherein the trending short videos are those whose popularity exceeds a preset index value;

[0149] Based on the video titles of various popular short videos, popular short videos that do not involve film and television content are filtered out to obtain multiple effective short videos, and each effective short video is divided into multiple effective segments;

[0150] Video fingerprinting technology is used to identify the original video clip to which each valid segment in each valid short video belongs, and the number of times the same segment in the same original video clip is used in the short video is counted.

[0151] Based on the number of times the same segment from the same original video is used in short videos, sample videos are selected from the original video and video-level tags are obtained for each sample video.

[0152] This disclosure also provides a block diagram of a device for detecting high-energy segments in film and television videos, the device comprising:

[0153] The video acquisition module is used to acquire the video to be detected and to divide the video into multiple video segments;

[0154] The feature extraction module is used to extract the visual and audio features of each video segment in the film and television video using a pre-trained encoder;

[0155] The high-energy detection module is used to output a high-energy score for each video segment in the film and television video based on the visual and audio features of each video segment in the film and television video using a high-energy detection model; the high-energy detection model is trained using the training method described above.

[0156] The selection module is used to identify high-energy segments based on video segments with high-energy scores exceeding a specified threshold.

[0157] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0158] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0159] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0160] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0161] Figure 5 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 5The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0162] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0163] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0164] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0165] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0166] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0167] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0168] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0169] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0170] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0171] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a high-energy detection model for detecting high-energy segments in film and television videos, characterized in that, include: Obtain a dataset that includes multiple sample videos of various film and television types and video-level tags for each sample video. Each sample video is divided into multiple sample segments, and the video-level tags are used to indicate whether there are high-energy segments in the sample video. Using the video-level labels of each sample video in the dataset, the initial detection model is trained in the first stage to obtain the intermediate detection model; The intermediate detection model is trained in the second stage using the segment-level labels of each sample segment in each sample video generated by the large language model, resulting in a trained high-energy detection model. The segment-level labels represent the high-energy level of the sample segments.

2. The method according to claim 1, characterized in that, The process of using video-level labels from each sample video in the dataset to train the initial detection model in the first stage, thereby obtaining an intermediate detection model, includes: For each sample video, a pre-trained encoder is used to extract the visual and audio features of each sample segment in each sample video; The initial detection model is used to generate a high-energy score for each sample segment in each sample video based on the visual and audio features of each sample segment. The high-energy score represents the probability that the sample segment is high-energy. For any given sample video, select the K highest-scoring sample segments from that sample video, and use the average of the high-scoring scores of these K sample segments as the video-level prediction result for that sample video. By utilizing the loss between the video-level labels of each sample video and the video-level prediction results of each sample video, the model parameters of the initial detection model are optimized to obtain an intermediate detection model.

3. The method according to claim 1 or 2, characterized in that, The segment-level labels generated by the large language model for each sample video are used to perform a second-stage training on the intermediate detection model to obtain a trained high-energy detection model, including: In the nth round of training, the large language model is used to generate the nth round segment-level label corresponding to each sample segment in each sample video based on the content description of each sample segment in each sample video and the high-energy mode of the (n-1th round). The high-energy mode includes high-energy feature descriptions under different film and television types. The content description is generated based on the sample segments using the multimodal large language model. 1≤n≤N, the high-energy mode of the 0th round is summarized as the initial high-energy mode, and N is the total number of training rounds. Based on the segment-level label of the nth round corresponding to each sample segment in each sample video, the detection model of the (n-1)th round is trained in multiple batches to obtain the nth round detection model, and the 0th round detection model is the intermediate detection model; Using the large language model, based on the high-energy score of each sample segment in each sample video output during the multi-batch iterative training, the content description of each sample segment and the corresponding film and television type, the nth round of high-energy mode is generated; Among them, the detection model obtained after the Nth round of training is the trained high-energy detection model.

4. The method according to claim 3, characterized in that, The process involves iteratively training the (n-1)th round detection model using the nth round segment-level labels corresponding to each sample segment in each sample video to obtain the nth round detection model, including: In the m-th batch of training in the n-th round, the (m-1)-th generation detection model is used to generate the prediction results of the m-th batch of videos based on the visual and audio features of each sample segment in the m-th batch of videos. The prediction results include the high-energy score of each sample segment in the m-th batch of videos, 1≤m≤M; the M batches of videos are obtained by dividing the sample videos in the dataset, and the 0th generation detection model is the (n-1)-th round detection model; The target loss is determined by using the prediction results of the m-th batch of videos, the (m-1)-th generation segment-level labels, and the n-th round segment-level labels, wherein the 0th generation segment-level labels are the n-th round segment-level labels; The model parameters of the (m-1)th generation detection model and the (m-1)th generation fragment-level labels are optimized using the target loss to obtain the m-th generation detection model and the m-th generation fragment-level labels; The Mth generation detection model obtained from the Mth batch of training in the nth round is the nth round detection model.

5. The method according to claim 4, characterized in that, The determination of the target loss using the prediction results of the m-th batch of videos, the (m-1)-th generation segment-level labels, and the n-th round of segment-level labels includes: Based on the prediction results of the m-th batch of videos and the segment-level labels of the (m-1)th generation, the segment classification loss is determined. The segment classification loss is used to make the prediction results output by the detection model as close as possible to the segment-level labels. Based on the (m-1)th generation fragment-level labels and the nth round fragment-level labels, a compatibility loss is calculated. The compatibility loss is used to ensure that the optimized fragment-level labels do not deviate excessively from the nth round fragment-level labels. Based on the prediction results of the m-th batch of videos, a sharpness loss is determined. The sharpness loss is used to encourage the high-energy score output by the detection model to approach the maximum or minimum extreme value, wherein the high-energy score output by the detection model is between the maximum and minimum extreme values. The target loss is determined based on the segment classification loss, the compatibility loss, and the sharpness loss.

6. The method according to claim 3, characterized in that, The process of generating the nth round high-energy mode using the large language model based on the high-energy score of each sample segment in each sample video, the content description of each sample segment, and the corresponding film and television type output during the multi-batch iterative training of the n-1th round detection model includes: Based on the high-energy scores of each sample segment in each sample video output during the multi-batch iterative training of the detection model in the (n-1)th round, multiple target sample segments with high-energy scores higher than a preset threshold are selected. Using the large language model, based on the content descriptions of the multiple target sample segments and their corresponding film and television types, the nth round of high-energy mode is generated.

7. The method according to claim 1, characterized in that, The acquisition of the dataset includes: Acquire multiple trending short videos, wherein the trending short videos are those whose popularity exceeds a preset index value; Based on the video titles of various popular short videos, popular short videos that do not involve film and television content are filtered out to obtain multiple effective short videos, and each effective short video is divided into multiple effective segments; Video fingerprinting technology is used to identify the original video clip to which each valid segment in each valid short video belongs, and the number of times the same segment in the same original video clip is used in the short video is counted. Based on the number of times the same segment from the same original video is used in short videos, sample videos are selected from the original video and video-level tags are obtained for each sample video.

8. A method for detecting high-energy segments in film and television videos, characterized in that, include: The video to be detected is acquired and then divided into multiple video segments. The visual and audio features of each video segment in the film and television video are extracted using a pre-trained encoder; The high-energy detection model is used to output a high-energy score for each video segment in the film and television video based on the visual and audio features of each video segment; the high-energy detection model is trained using the method described in any one of claims 1 to 8. Video segments with high energy scores above a specified threshold are selected as the detected high-energy segments.

9. A high-energy detection model training device for detecting high-energy segments in film and television videos, characterized in that, include: The acquisition module is used to acquire a dataset, which includes multiple sample videos of various film and television types and video-level tags for each sample video. Each sample video is divided into multiple sample segments, and the video-level tags are used to indicate whether there are high-energy segments in the sample video. The first-stage training module is used to train the initial detection model in the first stage using the video-level labels of each sample video in the dataset, so as to obtain the intermediate detection model. The second-stage training module is used to train the intermediate detection model using the segment-level labels of each sample segment in each sample video generated by the large language model, so as to obtain a trained high-energy detection model. The segment-level labels represent the high-energy level of the sample segments.

10. A device for detecting high-energy segments in film and television videos, characterized in that, include: The video acquisition module is used to acquire the video to be detected and to divide the video into multiple video segments; The feature extraction module is used to extract the visual and audio features of each video segment in the film and television video using a pre-trained encoder; The high-energy detection module is used to output a high-energy score for each video segment in the film and television video based on the visual and audio features of each video segment in the film and television video using a high-energy detection model. The high-energy detection model is trained using the method described in any one of claims 1 to 8; The high-energy determination module is used to select video segments with high-energy scores higher than a specified threshold as detected high-energy segments.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

12. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images