Video highlight detection method and device, readable storage medium and program product

Abstract

The invention relates to a video highlight detection method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a to-be-detected video, and segmenting the video according to an action picture in the video to obtain a plurality of video clips; generating a highlight description text of the video, and determining a first highlight detection result of each video clip according to the highlight description text; determining sound information of the video, and determining a second highlight detection result of each video clip according to the sound information in the video; and according to the first highlight detection result and the second highlight detection result of each video clip, determining the highlight clip in the video. By adopting the method, the accuracy of the determined highlight segment can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video highlight detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Video highlight detection refers to identifying and extracting the most attractive, informative, or emotionally impactful segments from long videos, such as goals in sports events, climax performances in concerts, key plot twists in movies, and peak interactions in live streams.

[0003] Existing video highlight detection methods are mainly based on Visual Language Models (VLMs), which use a single prompt to drive the VLM to directly output the start and end times of highlight segments from the video. However, due to the limitations of VLMs in video understanding, they suffer from inaccurate highlight segment identification. Summary of the Invention

[0004] Based on this, this application provides a video highlight detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can improve the accuracy of highlight detection.

[0005] On the one hand, this application provides a video specular detection method, including:

[0006] The video to be detected is acquired, and the video is segmented according to the action scenes in the video to obtain multiple video clips;

[0007] Generate highlight description text for the video, and determine the first highlight detection result for each video segment based on the highlight description text;

[0008] Determine the audio information of the video, and based on the audio information in the video, determine the second highlight detection result for each of the video segments;

[0009] The highlight segments in the video are determined based on the first highlight detection result and the second highlight detection result of each video segment.

[0010] In one embodiment, the step of segmenting the video based on action frames in the video to obtain multiple video segments includes:

[0011] Based on the temporal feature sequence of the video, predict the continuous actions in the video to obtain the temporal boundaries of the continuous actions;

[0012] The video is segmented according to the time boundaries of the continuous actions to obtain multiple video clips.

[0013] In one embodiment, predicting continuous actions in the video based on the temporal feature sequence of the video to obtain the temporal boundaries of the continuous actions includes:

[0014] The video is segmented according to a preset time interval to obtain segmented segments, and the visual features of each segment are extracted to obtain a temporal feature sequence.

[0015] The time-series feature sequence is subjected to self-attention processing to obtain a fused feature sequence;

[0016] Multiple anchor points are set for the fused feature sequence; each anchor point corresponds to at least one fused feature in the fused feature sequence;

[0017] For each anchor point, the action confidence, action category, and anchor point boundary information are determined based on the fusion features corresponding to the anchor point.

[0018] Based on the action confidence level corresponding to each of the anchor points, determine the action anchor point among the plurality of anchor points;

[0019] Based on the action category and anchor point boundary information corresponding to each of the aforementioned action anchor points, the continuous actions in the video and the time boundaries of the continuous actions are determined.

[0020] In one embodiment, generating the highlight description text of the video includes:

[0021] A visual language model is determined, and the video is input into the visual language model so that the visual language model generates video description text for the video;

[0022] A large language model is determined, and the video description text is input into the large language model so that the large language model outputs the highlight description text of the video.

[0023] In one embodiment, determining the first highlight detection result for each of the video segments based on the highlight description text includes:

[0024] The highlight description text is encoded to obtain text embedding features;

[0025] For each of the plurality of video segments, the video segment is encoded to obtain the visual embedding features of the video segment;

[0026] Determine the similarity between the visual embedding features and the text embedding features;

[0027] The first highlight detection result of the video segment is determined based on the similarity between the visual embedding features and the text embedding features.

[0028] In one embodiment, determining the second highlight detection result for each of the video segments based on the audio information in the video includes:

[0029] The target audio is obtained by extracting audio belonging to a preset category from the video;

[0030] Determine the energy spectrum of the target audio;

[0031] Based on the energy spectrum, the second highlight detection result for each of the video segments is determined.

[0032] In one embodiment, determining the second highlight detection result for each of the video segments based on the energy spectrum includes:

[0033] For each of the plurality of video segments, the peak and average sound energy of the video segment are determined based on the energy spectrum.

[0034] The second highlight detection result of the video segment is determined based on the peak sound energy and the average sound energy.

[0035] In one embodiment, the first highlight detection result includes a first score for the video clip to be a video highlight in the visual dimension; the second highlight detection result includes a second score for the video clip to be a video highlight in the audio dimension.

[0036] The step of determining the highlight segments in the video based on the first highlight detection result and the second highlight detection result of each of the video segments includes:

[0037] For each of the plurality of video segments, the first score and the second score of the video segment are fused to obtain the comprehensive score of the video segment belonging to the highlight video segment;

[0038] The highlight segments in the video are determined based on the comprehensive score of each video segment.

[0039] On the one hand, this application also provides a video highlight detection device, including:

[0040] The segmentation module is used to acquire the video to be detected and segment the video according to the action scenes in the video to obtain multiple video segments.

[0041] The detection result acquisition module is used to generate highlight description text for the video, and determine the first highlight detection result for each of the video segments based on the highlight description text; determine the audio information of the video, and determine the second highlight detection result for each of the video segments based on the audio information in the video;

[0042] The highlight segment determination module is used to determine the highlight segments in the video based on the first highlight detection result and the second highlight detection result of each video segment.

[0043] On the one hand, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0044] The video to be detected is acquired, and the video is segmented according to the action scenes in the video to obtain multiple video clips;

[0045] Generate highlight description text for the video, and determine the first highlight detection result for each video segment based on the highlight description text;

[0046] Determine the audio information of the video, and based on the audio information in the video, determine the second highlight detection result for each of the video segments;

[0047] The highlight segments in the video are determined based on the first highlight detection result and the second highlight detection result of each video segment.

[0048] On the one hand, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0049] The video to be detected is acquired, and the video is segmented according to the action scenes in the video to obtain multiple video clips;

[0050] Generate highlight description text for the video, and determine the first highlight detection result for each video segment based on the highlight description text;

[0051] Determine the audio information of the video, and based on the audio information in the video, determine the second highlight detection result for each of the video segments;

[0052] The highlight segments in the video are determined based on the first highlight detection result and the second highlight detection result of each video segment.

[0053] On the one hand, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0054] The video to be detected is acquired, and the video is segmented according to the action scenes in the video to obtain multiple video clips;

[0055] Generate highlight description text for the video, and determine the first highlight detection result for each video segment based on the highlight description text;

[0056] Determine the audio information of the video, and based on the audio information in the video, determine the second highlight detection result for each of the video segments;

[0057] The highlight segments in the video are determined based on the first highlight detection result and the second highlight detection result of each video segment.

[0058] The aforementioned video highlight detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product, by acquiring the video to be detected, can generate highlight description text for the video, and can segment the video according to the action scenes in the video to obtain multiple video segments. By generating highlight description text and video segments, highlight detection can be performed on each video segment according to the highlight description text to obtain the first highlight detection result of each video segment in the visual dimension. The audio information of the video can also be determined, and the second highlight detection result of each video segment in the audio dimension can be determined based on the audio information. By determining the first and second highlight detection results of each video segment, the first and second highlight detection results of each video segment can be combined to determine the highlight segments in the video. This application, by introducing highlight description text, compensates for the deficiencies of visual language models in understanding abstract highlight definitions and their limited ability to understand long videos, thereby improving the accuracy of the identified highlight segments based on highlight description text. Furthermore, this application not only relies on visual content but also combines audio information for highlight recognition, capturing highlight information in the video from multiple dimensions, which can further improve the accuracy of the identified highlight segments. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is an application environment diagram of the video highlight detection method in one embodiment;

[0061] Figure 2This is a flowchart illustrating a video highlight detection method in one embodiment;

[0062] Figure 3 This is a schematic diagram illustrating the determination of the time boundary of continuous actions in one embodiment;

[0063] Figure 4 This is a schematic diagram illustrating the generation of highlight description text in one embodiment;

[0064] Figure 5 This is a schematic diagram illustrating the generation of the first highlight detection result in one embodiment;

[0065] Figure 6 This is a schematic diagram illustrating the determination of the second highlight detection result in one embodiment;

[0066] Figure 7 This is the overall process for video highlight detection in one embodiment;

[0067] Figure 8 This is a structural block diagram of a video highlight detection device in one embodiment;

[0068] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0069] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0070] The video highlight detection method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on the cloud or other network servers. Both terminal 102 and server 104 can be used independently to execute the video highlight detection method provided in this embodiment. Terminal 102 and server 104 can also be used collaboratively to execute the video highlight detection method provided in this embodiment. Taking the example of terminal 102 and server 104 collaboratively executing the video highlight detection method provided in this embodiment, terminal 102 can acquire the video to be detected and send the video to server 104. Server 104 can generate highlight description text for the video and segment the video according to the continuity of actions to obtain multiple video segments. Server 104 can determine the first highlight detection result of each video segment in the visual dimension based on the highlight description text, and determine the second highlight detection result of each video segment in the audio dimension based on the sound in the video. Server 104 determines the highlight segments in the video based on the first and second highlight detection results of each video segment.

[0071] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0072] In one exemplary embodiment, such as Figure 2 As shown, a video specular highlight detection method is provided, and its application to a computer device is illustrated as an example. The computer device can provide... Figure 1 The terminal or server in the process. The video highlight detection method includes the following steps 202 to 208. Wherein:

[0073] Step 202: Obtain the video to be detected, and segment the video according to the action scenes in the video to obtain multiple video clips.

[0074] In this context, action footage in a video refers to visual images within the video that contain clearly defined dynamic behaviors, limb movements, or object displacements. The core characteristic is the presence of a perceptible state of motion, rather than a static scene or image (such as a pure landscape or a still close-up of a person). Action footage in a video can be either the active actions of a person or the passive movement of an object.

[0075] Optionally, the computer device can acquire the video to be inspected for highlights, identify continuous actions in the video based on the action frames, and segment the video according to these continuous actions to obtain multiple video clips. Continuous actions refer to actions that are sequential and of a certain duration. For example, a cooking video can be segmented into multiple clips such as "getting ingredients," "chopping vegetables," "stir-frying," and "plating" based on the continuity of the actions.

[0076] Optionally, a pre-trained action segmentation model can be used to identify continuous actions in a video and segment the video into multiple video clips based on these continuous actions. During the training of the action segmentation model, multiple video samples can be acquired, and each video sample can be labeled; for example, continuous action clips within the video samples can be labeled. The action segmentation model can then be trained using the video samples and their labeled annotations.

[0077] Step 204: Generate highlight description text for the video, and determine the first highlight detection result for each video segment based on the highlight description text.

[0078] Highlight description text is a textual summary and explanation of highlight segments in a video. For example, in a football match video, the highlight description text could be "A player receives a cross from a teammate, heads the ball into the goal inside the penalty area, and helps the team equalize the score." In addition to the textual summary and explanation of the highlight segment, the highlight description text can also include the time of the highlight segment. For example, in the example above, the highlight segment could be "In the 89th minute of the football match, a player receives a cross from a teammate, heads the ball into the goal inside the penalty area, and helps the team equalize the score; the focus of the image is on the moment of the goal." The first highlight detection result is the visual detection result of the video segment, reflecting the probability value that the video segment belongs to the video highlights.

[0079] Optionally, the computer device can acquire the video to be detected for highlights and generate highlight description text for the video. For example, the computer device can input the video into a visual language model, generate video description text for the video through the visual language model, and input the video description text into a large language model, which then outputs highlight description text. For each video segment among multiple video segments, the computer device can determine the probability value of the video segment belonging to video highlights based on the highlight description text, thereby obtaining the first highlight detection result for that video segment. For example, based on the probability value, a score for the video segment belonging to video highlights can be calculated, and this score can be used as the first highlight detection result for that video segment; or, the probability value can be directly used as the first highlight detection result for that video segment.

[0080] Optionally, since the highlight description text is a textual summary and explanation of the highlight segments in the video, it can be determined whether the video segment matches the description in the highlight description text. The higher the matching degree, the higher the probability value of the video segment belonging to the video highlight.

[0081] Optionally, video clips can be input into a visual language model, which outputs a clip description text. The clip description text is then matched with the highlight description text. The higher the matching degree, the higher the probability that the video clip belongs to the video highlight.

[0082] Step 206: Determine the audio information of the video, and based on the audio information in the video, determine the second highlight detection result for each video segment.

[0083] The second highlight detection result is the detection result of the video segment in the audio dimension, which reflects the probability value of the video segment belonging to the video highlight.

[0084] Optionally, audio information can be extracted from the video, and based on the extracted audio information, the probability value of each video segment belonging to a video highlight can be determined, thereby obtaining the second highlight detection result for each video segment. For example, based on the probability value of a video segment belonging to a highlight segment, a score for that video segment belonging to a video highlight can be calculated, and this score can be used as the second highlight detection result for that video segment; or, the probability value can be directly used as the second highlight detection result for that video segment.

[0085] Optionally, background noise can be removed from the video, and sounds belonging to a preset category can be extracted to obtain target audio. The probability value of each video segment belonging to a video highlight can then be determined based on the sound energy of the target audio. For example, human voices can be extracted from video segments, and the probability value of each video segment belonging to a video highlight can be determined based on the sound energy of the extracted human voices.

[0086] Optionally, for each video segment, based on the target audio, the higher the sound energy of the video segment, the greater the probability that the video segment belongs to a video highlight. Optionally, the average sound energy of the video segment can be calculated based on the target audio, and the probability that the video segment belongs to a video highlight can be determined based on the mapping relationship between the average sound energy and the probability value.

[0087] Step 208: Based on the first and second highlight detection results of each video segment, determine the highlight segments in the video.

[0088] Optionally, after obtaining the first and second highlight detection results for each video segment, for each video segment, since the first highlight detection result reflects the probability value of the video segment belonging to video highlights in the visual dimension, and the second highlight detection result reflects the probability value of the video segment belonging to video highlights in the audio dimension, the first and second highlight detection results of the video segment can be combined to determine the comprehensive detection result of the video segment. Thus, given the comprehensive detection results of each video segment, highlight segments in the video can be identified based on the comprehensive detection results of each video segment.

[0089] Optionally, the first highlight detection result can be the score of the video clip as a video highlight in the visual dimension; the second highlight detection result can be the score of the video clip as a video highlight in the audio dimension. Therefore, the first highlight detection result and the second highlight detection result can be weighted and fused. For example, the first highlight detection result and the second highlight detection result can be weighted and averaged to obtain the comprehensive score of the video clip, and this comprehensive score can be used as the comprehensive detection result.

[0090] Optionally, after determining the comprehensive score of each video segment, the video segments can be sorted in descending order of comprehensive score to obtain a video segment sequence, and the first preset number of video segments in the video segment sequence can be used as the highlight segments in the video.

[0091] Optionally, multiple video clips can be filtered according to preset duration conditions to obtain candidate video clips. These candidate video clips are then sorted in descending order of their comprehensive scores to obtain a video clip sequence. For example, video clips shorter than 3 seconds and those longer than 15 seconds can be removed from multiple video clips to obtain candidate video clips. By filtering multiple video clips according to preset duration conditions, the final selected highlight clips can better meet the user's needs in terms of duration.

[0092] In the aforementioned video highlight detection method, by acquiring the video to be detected, highlight description text for the video can be generated, and the video can be segmented based on the action scenes in the video to obtain multiple video segments. By generating highlight description text and video segments, highlight detection can be performed on each video segment based on the highlight description text to obtain the first highlight detection result of each video segment in the visual dimension. The audio information of the video can also be determined, and the second highlight detection result of each video segment in the audio dimension can be determined based on the audio information. By determining the first and second highlight detection results of each video segment, the first and second highlight detection results of each video segment can be combined to identify the highlight segments in the video. This application, by introducing highlight description text, compensates for the shortcomings of visual language models in understanding abstract highlight definitions and their limited ability to understand long videos, thereby improving the accuracy of the identified highlight segments based on highlight description text. Furthermore, this application not only relies on visual content but also combines audio information for highlight recognition, capturing highlight information in the video from multiple dimensions, which can further improve the accuracy of the identified highlight segments.

[0093] In an exemplary embodiment, step 202, which involves segmenting the video based on action frames to obtain multiple video segments, may include: predicting continuous actions in the video based on the temporal feature sequence of the video to obtain the temporal boundaries of the continuous actions; and segmenting the video based on the temporal boundaries of the continuous actions to obtain multiple video segments.

[0094] Optionally, highlight segments are generally based on temporal actions. Therefore, a temporal feature sequence of the video can be generated, and continuous actions in the video can be predicted based on this sequence to obtain the temporal boundaries of those actions. For example, in a cooking video, continuous actions can be predicted, determining the time boundary for "getting ingredients" as 00:00-00:14, the time boundary for "chopping vegetables" as 00:15-01:19, and so on. Once the temporal boundaries of the continuous actions are determined, the video can be segmented based on these boundaries to obtain multiple video segments. For example, in the above example, 00:00-00:14 can be considered as one video segment, and 00:15-01:19 as another.

[0095] Optionally, refer to Figure 3 , Figure 3 The diagram illustrates the determination of the temporal boundaries of continuous actions in one embodiment. Latent features of the input video can be extracted using an action segmentation model. These latent features are then fed into a Transformer-based backbone network within the action segmentation model to perform action boundary regression at each moment of the video, thereby obtaining the temporal boundaries of all continuous actions in the video.

[0096] In this embodiment, since highlight segments are generally based on actions, action extraction helps improve the accuracy of highlight moment start and end time location, thereby improving the accuracy of highlight detection.

[0097] In an exemplary embodiment, predicting continuous actions in a video based on its temporal feature sequence to obtain the temporal boundaries of the continuous actions includes: segmenting the video according to a preset time interval to obtain segmented segments, and extracting visual features from each segmented segment to obtain a temporal feature sequence; performing self-attention processing on the temporal feature sequence to obtain a fused feature sequence; setting multiple anchor points for the fused feature sequence; each anchor point corresponding to at least one fused feature in the fused feature sequence; for each anchor point, determining the action confidence, action category, and anchor point boundary information corresponding to the anchor point based on the fused feature corresponding to the anchor point; determining the action anchor point among the multiple anchor points based on the action confidence corresponding to each anchor point; and determining the continuous actions and the temporal boundaries of the continuous actions in the video based on the action category and anchor point boundary information corresponding to each action anchor point.

[0098] Optionally, the action segmentation model can segment the video according to a preset time interval (e.g., one segment every 16 frames) to obtain multiple segmented segments, and extract the visual features of each segment to obtain a temporal feature sequence F=[f1, f2, ..., fT] (T is the temporal length, and fT is the visual feature of the t-th segment). The feature sequence preserves the temporal order to reflect the continuity of the actions. Furthermore, self-attention processing can be applied to the temporal feature sequence, so that each visual feature becomes a fused feature that integrates all relevant feature information; these fused features constitute the fused feature sequence. By performing self-attention processing, each visual feature can focus on all other visual features in the temporal feature sequence, thereby understanding the continuity of actions. Furthermore, when a fused feature sequence is obtained, multiple anchor points can be set. Each anchor point serves as a candidate bounding box for selecting at least one segment in the video; multiple anchor points are used together to select the entire video without omission; each anchor point corresponds to at least one fused feature in the fused feature sequence. Multi-scale anchor points can be set; for example, in the case where each segment includes 16 video frames, short, medium, and long anchor points can be set. Specifically, a short anchor point corresponds to 5 fused features (selecting 5 × 16 = 80 frames in the video, approximately 3.3 seconds); a medium anchor point corresponds to 10 fused features (selecting 160 frames, approximately 6.7 seconds); and a long anchor point corresponds to 20 temporal features (selecting 320 frames, approximately 13.3 seconds).

[0099] Furthermore, the action segmentation model can determine whether the segmented fragment selected by the anchor point is an action fragment based on the fusion features corresponding to the anchor point, thereby obtaining the action confidence corresponding to that anchor point. For example, if the anchor point selects the first and second segmented fragments, then the anchor point corresponds to the fusion features... Then the fusion characteristics can be determined. The probability value of a segment corresponding to a point being an action segment is used as the action confidence level corresponding to that point. Furthermore, the action segmentation model can also determine the action category to which the segment selected by the anchor point belongs based on the fusion features corresponding to the anchor point.

[0100] The motion segmentation model can also determine the boundary offset corresponding to the anchor point. By adjusting the boundary offset, the anchor point's boundary is obtained, including its start and end positions. Since the anchor point corresponds to a fixed video segment, but the boundaries of the actual motion often do not completely coincide with the anchor point, the motion segmentation model can predict the anchor point's boundary offset, allowing the adjusted start and end positions of the anchor point to better match the actual motion boundaries. For example, when the anchor point corresponds to a fusion feature... In this case, the original starting position of the anchor point is the 17th frame. The action segmentation model outputs "starting offset = -0.5", which means that the starting position needs to be moved forward by 0.5 fusion features (corresponding to 8 video frames). After correction, the starting position is the 9th frame in the video.

[0101] Furthermore, once the action confidence, action category, and anchor point boundary information for each anchor point are obtained, action anchor points among multiple anchor points can be determined based on the action confidence of each anchor point. Here, an action anchor point refers to the anchor point where the selected segment is an action segment. Based on the action category and anchor point boundary information for each action anchor point, continuous actions and their temporal boundaries in the video are determined. For example, based on the anchor point boundary information, continuous actions in the video are determined, where the start and end positions of the continuous action are consistent with the start and end positions in the anchor point boundary information, and the action category corresponding to that anchor point is taken as the action category to which the continuous action belongs.

[0102] In this embodiment, anchor points are used to determine the continuous actions and their time boundaries in the video, making the determined continuous actions and their time boundaries more accurate.

[0103] In an exemplary embodiment, generating the highlight description text of the video in step 204 may include: determining a visual language model and inputting the video into the visual language model so that the visual language model generates the video description text of the video; determining a large language model and inputting the video description text into the large language model so that the large language model outputs the highlight description text of the video.

[0104] Among them, the Vision-Language Model (VLM) is a multimodal artificial intelligence model that integrates computer vision and natural language processing capabilities. VLMs can learn the relationship between text data and visual data such as images or videos, thus generating text based on visual input. The Large Language Model (LLM) is an artificial intelligence model based on deep learning technology, with the core capability of "understanding and generating human language." It includes a complex network structure and a large number of model parameters, and can achieve language interaction and content generation by learning the grammar, semantics, logic, and even cultural background of a language.

[0105] The video and first-model prompts can be input into the visual language model, which will then output a video description text. The first-model prompts are the prompts used to guide the visual language model in outputting the video description text. The video description text is a detailed textual description, explanation, or supplement to the video content (including visuals, sound, plot, core information, etc.).

[0106] Furthermore, given the video description text, the video description text and the second model prompt information can be input into the large language model, which then outputs the highlight description text. The second model prompt information refers to the prompt words used to guide the large language model in outputting the highlight description text.

[0107] Optionally, refer to Figure 4 When a video is available, the video and the first model prompt "describe the scene in the video in detail" can be input into the visual language model, which will output a video description text. The video description text and the second model prompt "what is the most exciting scene in this description?" can be input into the large language model, which will output a highlight description text. Figure 4 A schematic diagram illustrating the generation of highlight descriptive text in one embodiment is shown.

[0108] In this embodiment, by first generating video description text (covering the entire video content) and then extracting highlight descriptions from it, the video description text can accurately anchor segments strongly related to video highlights, rather than mistakenly treating irrelevant details as highlights, thereby ensuring the accuracy of highlights: avoiding deviation from the core of the video, reducing misalignment of highlights, and thus improving the accuracy of highlight description text.

[0109] In an exemplary embodiment, step 206, for each of the plurality of video segments, determining the first highlight detection result of the video segment based on the highlight description text, includes: encoding the highlight description text to obtain text embedding features; for each of the plurality of video segments, encoding the video segment to obtain visual embedding features of the video segment; determining the similarity between the visual embedding features and the text embedding features; and determining the first highlight detection result of the video segment based on the similarity between the visual embedding features and the text embedding features.

[0110] Optionally, the highlight description text can be encoded to obtain text embedding features. For each video segment, the video segment can be encoded to obtain visual embedding features for that video segment. For each visual embedding feature, the cosine similarity between the visual embedding feature and the text embedding feature is calculated. The cosine similarity is used as the first probability value that the video segment corresponding to the visual embedding feature belongs to the video highlight. Based on the first probability value, the first highlight detection result of the video segment is determined.

[0111] Optionally, the Euclidean distance or Manhattan distance between the visual embedding features and the text embedding features is also calculated. The similarity between the visual embedding features and the text embedding features is obtained by calculating the Euclidean distance or Manhattan distance between the visual embedding features and the text embedding features.

[0112] Optionally, refer to Figure 5 It allows setting both a visual encoder and a text encoder. The text encoder encodes the highlight description text to obtain text embedding features, while the visual encoder encodes the video clips to obtain visual embedding features. By obtaining the text embedding features and the visual embedding features, the similarity between the text embedding features and the visual embedding features can be calculated to obtain the first highlight detection result. Figure 5 A schematic diagram illustrating the generation of the first highlight detection result in one embodiment is shown.

[0113] Optionally, the visual encoder may use models such as ResNet (a residual neural network), ViT (VisionTransformer, a computer vision model based on the Transformer architecture), or their variants. The text encoder may use models such as BERT (Bidirectional Encoder Representations from Transformers, a bidirectional pre-trained language model based on the Transformer architecture), Vicuna (an open-source large language model), or their variants.

[0114] In this embodiment, by converting the video into highlight description text, the video highlights can possess clear semantic standards. By performing similarity matching between the visual embedding features extracted from the video clip and the text embedding features extracted from the highlight description text, the probability value of the video clip belonging to the video highlight can be determined based on the clear semantic standards, thereby improving the accuracy of the first highlight detection result.

[0115] Furthermore, traditional highlight recognition based on purely visual features often suffers from a "black box problem," failing to explain "why this video clip is a highlight," making it difficult to identify errors or adjust standards. However, determining the first highlight detection result based on the similarity between visual embedded features and text embedded features makes this result interpretable, facilitating manual verification or model optimization.

[0116] In an exemplary embodiment, step 208 includes: extracting audio belonging to a preset category from the video to obtain target audio; determining the audio segment corresponding to each video segment based on the target audio; and for each audio segment, determining the second highlight detection result of the corresponding video segment based on the sound energy of the audio segment.

[0117] Optionally, since audio information is an important reference indicator for video highlights—for example, highlights are usually accompanied by loud cheers, shouts, and screams from the surrounding audience—audio belonging to a preset category can be extracted from the video and used as the target audio. The preset category can be freely set according to needs; for example, the preset category could be human voice. Furthermore, after obtaining the target audio, it can be divided according to the playback time of each video segment to obtain the corresponding audio segment for each video segment. Once the corresponding audio segments for each video segment are obtained, the second highlight detection result for the corresponding video segment can be determined based on the sound energy of the audio segments. For example, for each video segment, the higher the sound energy of the segment, the higher the probability that the video segment belongs to a video highlight, and thus the higher the second score for the video segment belonging to a highlight video segment in the audio dimension.

[0118] Optionally, refer to Figure 6 , Figure 6 A schematic diagram illustrating the determination of the second highlight detection result in one embodiment is shown. When a video is acquired, the audio in the video can be separated into human voices to obtain the target audio. An energy spectrum is calculated for the target audio to obtain its energy spectrum. Based on the energy spectrum of the target audio, the sound energy of each video segment is calculated. The probability value of a video segment belonging to a highlight is determined based on the sound energy. Then, based on the probability value, a second score is obtained indicating that the video segment belongs to a highlight video segment in the audio dimension. This second score is used as the second highlight detection result for the video segment.

[0119] Optionally, the audio in the video can be converted into a spectrogram, and the U-Net architecture can be used to downsample the spectrogram to compress its dimensionality through convolutional and pooling layers, extracting high-frequency (detail) and low-frequency (global) features, and capturing the spectral differences between the audio of a preset category and the background audio. The features output from the downsampling stage are then upsampled using the U-Net architecture to restore the spectrogram dimension through deconvolutional layers, accurately locating the spectral region corresponding to the audio of the preset category. Based on the spectral region corresponding to the audio of the preset category, the audio of the preset category can be extracted from the video's audio to obtain the target audio.

[0120] In this embodiment, the highlight moments in the video may not be accompanied by strong visual changes, but they often transmit core signals through sound. By identifying video highlights through the sound (audio signal) in the video, the limitations of pure visual analysis can be compensated for by using sound, making the identification of video highlights more accurate.

[0121] In an exemplary embodiment, determining the second highlight detection result of the corresponding video segment based on the sound energy of the audio segment includes: dividing the audio segment into multiple audio sub-segments according to a preset duration; determining the sound energy value of each audio sub-segment; determining the highlight moment probability value corresponding to each audio sub-segment based on the sound energy value of each audio sub-segment; and determining the second highlight detection result of the video segment corresponding to the audio segment based on the highlight moment probability value corresponding to each audio sub-segment.

[0122] Optionally, the sound energy values ​​within preset time intervals in the audio segment can be determined; that is, the audio segment can be divided according to a preset duration to obtain multiple audio sub-segments; and the sound energy value of each audio sub-segment can be determined. Once the sound energy value of each audio sub-segment is determined, each sound energy value can be normalized according to a preset sound energy threshold to obtain the probability value of the highlight moment corresponding to each audio sub-segment. The average of each highlight probability value is calculated to obtain the second score of the video segment corresponding to the audio segment as belonging to the video highlight, and this second score is used as the second highlight detection result.

[0123] Optionally, for an audio segment, let the sound amplitude signal be y[n] and the sampling frequency be r, then the sound energy value within time t is:

[0124]

[0125] Normalizing E[t] using the statistically obtained sound energy threshold e yields the probability value of the highlight moment:

[0126]

[0127] The mean value of P[t] within the audio segment is calculated to obtain the second highlight detection result of the corresponding video segment.

[0128] In this embodiment, the second highlight detection result of the video segment is determined by combining multiple highlight moment probability values, so that the second highlight detection result can be more accurate.

[0129] In an exemplary embodiment, the first highlight detection result includes a first score for the video segment belonging to video highlights in the visual dimension; the second highlight detection result includes a second score for the video segment belonging to video highlights in the audio dimension; step 208 includes: for each video segment among multiple video segments, fusing the first score and the second score of the video segment to obtain a comprehensive score for the video segment belonging to highlight video segments; and determining the highlight segments in the video based on the comprehensive scores of each video segment.

[0130] Optionally, for each of the multiple video segments, the first score and the second score can be weighted and summed to obtain the overall score for that video segment. Once the overall scores for each video segment are obtained, the highlight segments in the video can be determined based on these scores. For example, the video segment with the highest overall score can be designated as the highlight segment in the video.

[0131] In this embodiment, the highlight segments in the video are determined by combining the first score of video highlights in the visual dimension and the second score of video highlights in the audio dimension, so that the determined highlight segments can be more accurate.

[0132] In one exemplary embodiment, reference Figure 7 , Figure 7This document illustrates the overall process of video highlight detection in one embodiment. The overall process comprises three main branches: visual semantics, action proposal, and sound amplitude. First, the action proposal module segments the video into several video segments based on the continuity of video actions, allowing subsequent algorithms to determine which video segment belongs to a highlight. The determination of highlight segments includes two dimensions: visual semantics and sound amplitude. In the visual semantics branch, the video is first input into a visual language model to generate detailed video description text. Then, the video description text is input into a large language model, utilizing its semantic understanding capabilities to extract highlight description text. Finally, the highlight description text is used to search and compare each video segment segmented by the action proposal branch, obtaining the first score for each video segment belonging to a highlight moment in the visual dimension. In the sound amplitude branch, background sound is first removed from the video to extract the human voice portion. Then, the human voice energy in each video segment is calculated to obtain the second score for that video segment belonging to a highlight moment in the audio dimension. Finally, the scores from the visual and sound dimensions are fused, and after filtering, the final highlight segments are obtained.

[0133] This application addresses the shortcomings of visual language models by introducing a large language model. To overcome the limitations of visual language models in understanding the abstract definition of highlight moments and their ability to process long videos, this application proposes using a large language model as a priori. Leveraging its stronger semantic understanding and prior knowledge, this approach compensates for the visual language model's deficiencies in understanding abstract highlight definitions and its limited ability to process long videos. This allows the visual language model to move beyond simply recognizing highlights in specific, previously trained scenes and better handle unseen scenarios, understanding the nuances of highlights in different contexts.

[0134] This application adopts a two-stage text processing workflow: a two-stage mode of generating detailed descriptions using a visual language model and extracting highlights using a large language model. First, a complete description of the video content is generated through a visual language model, and then a large language model summarizes the description of highlight moments from the text, replacing the single mode of directly outputting the start and end times of highlights by the visual language model, thereby improving the understanding of the overall video content and the accuracy of highlight moment recognition.

[0135] This application optimizes positioning accuracy by combining temporal action proposal: it introduces a temporal action proposal method to segment the original video based on action continuity, providing a foundation for accurate positioning of highlight segments and improving positioning accuracy. Furthermore, by leveraging temporal action proposal and video text retrieval methods, the video is first initially segmented based on action continuity, and then highlight description text is used to accurately match video segments, making the positioning of highlight segments more precise and reliable.

[0136] This application provides a multimodal fusion detection method: in addition to visual content, it also combines the sound energy spectrum of the audio channel for highlight moment identification. By using audio information to assist visual detection, it captures highlight information in the video from multiple dimensions, making the detection results more comprehensive and more in line with people's perception of highlight moments in real scenes, thereby enhancing the comprehensiveness and robustness of highlight detection.

[0137] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0138] Based on the same inventive concept, this application also provides a video highlight detection apparatus for implementing the video highlight detection method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more video highlight detection apparatus embodiments provided below can be found in the limitations of the video highlight detection method described above, and will not be repeated here.

[0139] In one exemplary embodiment, such as Figure 8 As shown, a video highlight detection device 800 is provided, including: a segmentation module 802, a detection result acquisition module 804, and a highlight segment determination module 806, wherein:

[0140] The segmentation module 802 is used to acquire the video to be detected and segment the video according to the action scenes in the video to obtain multiple video segments.

[0141] The detection result acquisition module 804 is used to generate highlight description text for the video, and determine the first highlight detection result for each video segment based on the highlight description text; determine the audio information of the video, and determine the second highlight detection result for each video segment based on the audio information in the video;

[0142] The highlight segment determination module 806 is used to determine the highlight segments in the video based on the first highlight detection result and the second highlight detection result of each video segment.

[0143] In an exemplary embodiment, the segmentation module 802 is further configured to predict continuous actions in the video based on the temporal feature sequence of the video to obtain the temporal boundaries of the continuous actions; and to segment the video based on the temporal boundaries of the continuous actions to obtain multiple video segments.

[0144] In an exemplary embodiment, the segmentation module 802 is further configured to segment the video according to a preset time interval to obtain segmented segments, and extract the visual features of each segmented segment to obtain a temporal feature sequence; perform self-attention processing on the temporal feature sequence to obtain a fusion feature sequence; set multiple anchor points for the fusion feature sequence; each anchor point corresponds to at least one fusion feature in the fusion feature sequence; for each anchor point, determine the action confidence, action category, and anchor point boundary information corresponding to the anchor point based on the fusion feature corresponding to the anchor point; determine the action anchor point among the multiple anchor points based on the action confidence corresponding to each anchor point; and determine the continuous actions and the temporal boundaries of the continuous actions in the video based on the action category and anchor point boundary information corresponding to each action anchor point.

[0145] In an exemplary embodiment, the detection result acquisition module 804 is further configured to determine a visual language model and input the video into the visual language model so that the visual language model generates video description text for the video; determine a large language model and input the video description text into the large language model so that the large language model outputs highlight description text for the video.

[0146] In an exemplary embodiment, the detection result acquisition module 804 is further configured to encode the highlight description text to obtain text embedding features; for each video segment in a plurality of video segments, encode the video segment to obtain visual embedding features of the video segment; determine the similarity between the visual embedding features and the text embedding features; and determine the first highlight detection result of the video segment based on the similarity between the visual embedding features and the text embedding features.

[0147] In an exemplary embodiment, the detection result acquisition module 804 is further configured to extract audio belonging to a preset category from the video to obtain target audio; determine the energy spectrum of the target audio; and determine the second highlight detection result of each video segment based on the energy spectrum.

[0148] In an exemplary embodiment, the detection result acquisition module 804 is further configured to, for each of the multiple video segments, determine the peak and average sound energy of the video segment based on the energy spectrum; and determine the second highlight detection result of the video segment based on the peak and average sound energy.

[0149] In an exemplary embodiment, the first highlight detection result includes a first score for the video segment belonging to video highlights in the visual dimension; the second highlight detection result includes a second score for the video segment belonging to video highlights in the audio dimension; the highlight segment determination module 806 is further configured to fuse the first score and the second score of each video segment among multiple video segments to obtain a comprehensive score of the video segment belonging to highlight video segments; and determine the highlight segments in the video based on the comprehensive scores of each video segment.

[0150] Each module in the aforementioned video highlight detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0151] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores video highlight detection data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a video highlight detection method.

[0152] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0153] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0154] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0155] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0156] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0157] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0158] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0159] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting video highlights, characterized in that, The method includes: The video to be detected is acquired, and the video is segmented according to the action scenes in the video to obtain multiple video clips; Generate highlight description text for the video, and determine the first highlight detection result for each video segment based on the highlight description text; Determine the audio information of the video, and based on the audio information in the video, determine the second highlight detection result for each of the video segments; The highlight segments in the video are determined based on the first highlight detection result and the second highlight detection result of each video segment.

2. The method according to claim 1, characterized in that, The step of segmenting the video based on action frames in the video to obtain multiple video clips includes: Based on the temporal feature sequence of the video, predict the continuous actions in the video to obtain the temporal boundaries of the continuous actions; The video is segmented according to the time boundaries of the continuous actions to obtain multiple video clips.

3. The method according to claim 2, characterized in that, The step of predicting continuous actions in the video based on the temporal feature sequence of the video to obtain the temporal boundaries of the continuous actions includes: The video is segmented according to a preset time interval to obtain segmented segments, and the visual features of each segment are extracted to obtain a temporal feature sequence. The time-series feature sequence is subjected to self-attention processing to obtain a fused feature sequence; Multiple anchor points are set for the fused feature sequence; each anchor point corresponds to at least one fused feature in the fused feature sequence; For each anchor point, the action confidence, action category, and anchor point boundary information are determined based on the fusion features corresponding to the anchor point. Based on the action confidence level corresponding to each of the anchor points, determine the action anchor point among the plurality of anchor points; Based on the action category and anchor point boundary information corresponding to each of the aforementioned action anchor points, the continuous actions in the video and the time boundaries of the continuous actions are determined.

4. The method according to claim 1, characterized in that, The generated highlight description text for the video includes: A visual language model is determined, and the video is input into the visual language model so that the visual language model generates video description text for the video; A large language model is determined, and the video description text is input into the large language model so that the large language model outputs the highlight description text of the video.

5. The method according to claim 1, characterized in that, The step of determining the first highlight detection result for each of the video segments based on the highlight description text includes: The highlight description text is encoded to obtain text embedding features; For each of the plurality of video segments, the video segment is encoded to obtain the visual embedding features of the video segment; Determine the similarity between the visual embedding features and the text embedding features; The first highlight detection result of the video segment is determined based on the similarity between the visual embedding features and the text embedding features.

6. The method according to claim 1, characterized in that, The step of determining the second highlight detection result for each video segment based on the audio information in the video includes: The target audio is obtained by extracting audio belonging to a preset category from the video; Based on the target audio, determine the audio segment corresponding to each of the video segments; For each audio segment, the second highlight detection result of the corresponding video segment is determined based on the sound energy of the audio segment.

7. The method according to claim 6, characterized in that, The step of determining the second highlight detection result of the corresponding video segment based on the sound energy of the audio segment includes: The audio segment is divided according to a preset duration to obtain multiple audio sub-segments; Determine the sound energy value of each audio sub-segment, and based on the sound energy value of each audio sub-segment, determine the probability value of the highlight moment corresponding to each audio sub-segment; Based on the probability value of the highlight moment corresponding to each of the audio sub-segments, the second highlight detection result of the video segment corresponding to the audio segment is determined.

8. The method according to claim 1, characterized in that, The first highlight detection result includes a first score for the video clip as a video highlight in the visual dimension; the second highlight detection result includes a second score for the video clip as a video highlight in the audio dimension. The step of determining the highlight segments in the video based on the first highlight detection result and the second highlight detection result of each of the video segments includes: For each of the plurality of video segments, the first score and the second score of the video segment are fused to obtain the comprehensive score of the video segment belonging to the highlight video segment; The highlight segments in the video are determined based on the comprehensive score of each video segment.

9. A video highlight detection device, characterized in that, The device includes: The segmentation module is used to acquire the video to be detected and segment the video according to the action scenes in the video to obtain multiple video segments. The detection result acquisition module is used to generate highlight description text for the video, and determine the first highlight detection result for each of the video segments based on the highlight description text; determine the audio information of the video, and determine the second highlight detection result for each of the video segments based on the audio information in the video; The highlight segment determination module is used to determine the highlight segments in the video based on the first highlight detection result and the second highlight detection result of each video segment.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images