Video outline generation method, device, computer equipment and medium

Abstract

The present application relates to a method, apparatus, computer equipment and medium for generating a video outline. The method includes: obtaining an initial video and determining a key video segment in the initial video; extracting an audio feature vector of the initial video and performing emotional feature detection on the audio feature vector, determining a basic audio segment in the audio feature vector based on the result of the emotional feature detection, detecting the rhythm intensity and audio short-time energy in the basic audio segment, and determining an audio highlight segment based on the rhythm intensity detection result and the audio short-time energy detection result; determining at least one video highlight segment in the initial video based on the video key segment and the audio highlight segment; screening out a target video highlight segment from the video highlight segment based on the motion trajectory of the moving target in the video highlight segment, and generating a video outline based on the target video highlight segment. High-quality video outlines can be generated through the present application.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, apparatus, computer equipment, and medium for generating a video outline. Background Art

[0002] With the development of the digital information age, digital video content has experienced explosive growth. Video summarization technology analyzes and processes the structure and content of the video, extracts representative video frames or video clips from the original video, and arranges and combines these meaningful video frames or clips to form a shorter and more compact summary video. This summary video contains a large amount of semantic information in the original video and can express the meaning and emotion of the original video to a certain extent.

[0003] However, the summary video extracted by the current video clip extraction technology is still relatively redundant and has low computational efficiency, and cannot effectively reflect the key information in the original video. Furthermore, the summary video generally has relatively chaotic logic and time.

[0004] Currently, there is no effective solution to the problem of how to efficiently extract key segments from an original video and generate a video segment that is more readable and enjoyable in the existing technology. Summary of the Invention

[0005] Based on this, it is necessary to provide a video outline generation method, device, computer equipment and medium to address the above technical problems.

[0006] In a first aspect, the present application provides a method for generating a video outline. The method comprises:

[0007] Obtaining an initial video and determining a key video segment in the initial video;

[0008] Extracting an audio feature vector from the initial video and performing emotion feature detection on the audio feature vector; determining a basic audio segment in the audio feature vector based on the results of the emotion feature detection; detecting the rhythm intensity and audio short-time energy in the basic audio segment; and jointly determining an audio highlight segment based on the rhythm intensity detection results and the audio short-time energy detection results;

[0009] Determining at least one video highlight segment in the initial video based on the video key segment and the audio highlight segment;

[0010] Based on the motion trajectory of the moving target in the video highlight clips, the target video highlight clips are screened out from the video highlight clips, and the video outline is generated based on the target video highlight clips.

[0011] In one embodiment, detecting the rhythm intensity and audio short-time energy in a basic audio segment, and jointly determining an audio highlight segment based on the rhythm intensity detection result and the audio short-time energy detection result, includes:

[0012] Performing audio short-time energy detection on the basic audio clip, calculating the root mean square of the audio short-time energy detection results, and determining the volume of each frame of audio in the basic audio clip based on the root mean square calculation results;

[0013] Determine the rhythm intensity of each frame of audio according to the difference between the spectral amplitude of each frame of audio in the basic audio segment and the spectral amplitude of the historical frame;

[0014] Perform a fusion weighted calculation on the volume of each frame of audio and the rhythm intensity of each frame of audio to obtain a highlight score for each frame of audio in the basic audio clip;

[0015] An audio highlight segment is obtained based on all audio frames whose highlight scores are greater than a preset score threshold.

[0016] In one embodiment, determining a key video segment in an initial video includes:

[0017] Obtaining video key frames in the initial video, and dividing the video material in the initial video into multiple video segments based on the video key frames;

[0018] All video clips are traversed, and the key video clips are determined based on the video feature vectors of each video clip.

[0019] In one embodiment, determining at least one video highlight segment in the initial video based on the video key segment and the audio highlight segment includes:

[0020] In the time dimension, based on the timestamps corresponding to the video key segments and the timestamps corresponding to the audio highlight segments, the initial video highlight segments are extracted from the initial video;

[0021] Fusing the image feature vector and the audio feature vector corresponding to the highlight segment of the initial video to obtain a fused feature vector of the highlight segment of the initial video;

[0022] Obtain the prompt words corresponding to the video outline and extract the text feature vector of the prompt words;

[0023] The text feature vector is matched with the fused feature vector for similarity to obtain the similarity corresponding to each fused feature vector, and the video highlight segment is determined based on the fused feature vector whose similarity is greater than a preset similarity threshold.

[0024] In one embodiment, extracting an initial video highlight segment from an initial video based on a timestamp corresponding to a video key segment and a timestamp corresponding to an audio highlight segment includes:

[0025] Restore the timestamps corresponding to the audio highlight clips to the timeline of the original video to obtain the audio highlight index corresponding to each audio highlight clip;

[0026] Restore the timestamps corresponding to the key segments of the video to the timeline of the original video to obtain the video key frame index corresponding to each key segment of the video;

[0027] At least one initial video highlight segment in the initial video is obtained based on the audio highlight index and the video key frame index.

[0028] In one embodiment, selecting a target video highlight segment from the video highlight segment based on a motion trajectory of a moving target in the video highlight segment includes:

[0029] Extract the motion trajectory of the moving target in each video highlight segment;

[0030] Performing anomaly detection on each motion trajectory, and determining a first motion trajectory among the motion trajectories based on the anomaly detection result; wherein the first motion trajectory includes a plurality of first sub-trajectories;

[0031] detecting a moving target in the first motion trajectory, and rearranging first sub-trajectories included in the first motion trajectory based on the moving target to obtain at least one second motion trajectory, wherein each second motion trajectory corresponds to a moving target;

[0032] Calculating a score for the second motion trajectory based on a dynamic programming algorithm, and determining a motion trajectory with a score greater than a preset second score threshold as a target motion trajectory; wherein the score calculation for the second motion trajectory is completed based on the video highlight score, the audio highlight score, and the motion trajectory highlight score of the second motion trajectory;

[0033] Based on the target motion trajectory, the corresponding target video highlight segment in the initial video is determined.

[0034] In one embodiment, determining a corresponding target video highlight segment in an initial video based on the target motion trajectory includes:

[0035] Determine target sub-motion trajectories in the target motion trajectory, rearrange the target sub-motion trajectories in chronological order, and obtain third motion trajectories that correspond one-to-one with the target sub-motion trajectories;

[0036] A target video highlight segment corresponding to the initial video is obtained based on the third motion trajectory.

[0037] In a second aspect, the present application further provides a video outline generation device. The device comprises:

[0038] An acquisition module, configured to acquire an initial video and determine a key video segment in the initial video;

[0039] a calculation module for extracting an audio feature vector from an initial video, performing emotion feature detection on the audio feature vector, determining a basic audio segment in the audio feature vector based on the results of the emotion feature detection, detecting rhythm intensity and audio short-time energy in the basic audio segment, and jointly determining an audio highlight segment based on the rhythm intensity detection results and the audio short-time energy detection results; and determining at least one video highlight segment in the initial video by integrating the video key segments and the audio highlight segments;

[0040] A generation module is used to filter out target video highlight segments from the video highlight segments based on the motion trajectory of the moving target in the video highlight segments, and generate a video outline based on the target video highlight segments, wherein the video outline represents the motion trajectory of the moving target in the target video highlight segments.

[0041] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0042] Obtaining an initial video and determining a key video segment in the initial video;

[0043] Extracting an audio feature vector from the initial video and performing emotion feature detection on the audio feature vector; determining a basic audio segment in the audio feature vector based on the results of the emotion feature detection; detecting the rhythm intensity and audio short-time energy in the basic audio segment; and jointly determining an audio highlight segment based on the rhythm intensity detection results and the audio short-time energy detection results;

[0044] Determining at least one video highlight segment in the initial video based on the video key segment and the audio highlight segment;

[0045] Based on the motion trajectory of the moving target in the video highlight clips, the target video highlight clips are screened out from the video highlight clips, and the video outline is generated based on the target video highlight clips.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0047] Obtaining an initial video and determining a key video segment in the initial video;

[0048] Extracting an audio feature vector from the initial video and performing emotion feature detection on the audio feature vector; determining a basic audio segment in the audio feature vector based on the results of the emotion feature detection; detecting the rhythm intensity and audio short-time energy in the basic audio segment; and jointly determining an audio highlight segment based on the rhythm intensity detection results and the audio short-time energy detection results;

[0049] Determining at least one video highlight segment in the initial video based on the video key segment and the audio highlight segment;

[0050] Based on the motion trajectory of the moving target in the video highlight clips, the target video highlight clips are screened out from the video highlight clips, and the video outline is generated based on the target video highlight clips.

[0051] The above-mentioned method, device, computer equipment and medium for generating a video outline extract key video segments based on the initial video, and determine audio highlight segments based on emotion feature detection, rhythm intensity detection and audio short-time energy detection. Based on the video key segments and audio highlight segments, multiple video highlight segments in the initial video are determined. Finally, based on the motion trajectory of the moving target in the video highlight segments, the target video highlight segments are screened out from the video highlight segments, and a video outline is generated. The video outline generated by this application ensures the refinement of the information, as well as the comprehensiveness and readability of the video outline. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A diagram illustrating an application environment of a method for generating a video outline in one embodiment;

[0053] Figure 2 A schematic diagram of a flow chart of a method for generating a video outline in one embodiment;

[0054] Figure 3 A schematic diagram of a process for generating a fused feature vector in one embodiment;

[0055] Figure 4 A schematic flow chart of a method for generating a video outline in a preferred embodiment;

[0056] Figure 5 is a structural block diagram of a video outline generating device in one embodiment;

[0057] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0059] The video outline generation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Obtain the initial video, and determine the key video segments in the initial video, extract the audio feature vector in the initial video, and perform emotional feature detection on the audio feature vector, determine the basic audio segment in the audio feature vector based on the result of the emotional feature detection, detect the rhythm intensity and audio short-time energy in the basic audio segment, and jointly determine the audio highlight segment based on the rhythm intensity detection result and the audio short-time energy detection result, and determine at least one video highlight segment in the initial video by integrating the video key segment and the audio highlight segment; based on the motion trajectory of the moving target in the video highlight segment, screen out the target video highlight segment from the video highlight segment, and generate a video outline based on the target video highlight segment, wherein the video outline characterizes the motion trajectory of the moving target in the target video highlight segment. Terminal 102 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0060] In one embodiment, Figure 2 As shown, a video outline generation method is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the server in the example:

[0061] Step S210: obtaining an initial video and determining key video segments in the initial video.

[0062] Specifically, in this application, the video outline refers to extracting some core segments from the initial video and rearranging the key target motion trajectories in the core segments according to a certain method and a certain logic to obtain a video outline. The video outline carries the key information in the original video and the important visual information that the user wants to obtain. Compared with the existing video summary extraction technology, the video outline extraction technology in this application can be based on the key segments in the original video, combined with target tracking and video content reorganization technology, to analyze and reorganize the extracted video materials, and finally obtain a video segment with high readability, viewing value, and more compact content, namely the above-mentioned video outline.

[0063] The key video clip refers to the key, core visual material (only visuals, no audio) in the initial video. This can be one or more video clips (only visuals, no audio) or one or more images. Extracting the key video clip from the initial video involves various methods. In some embodiments, the video material from the initial video can be first extracted, then segmented, and each video clip can be tested. For example, the video clip can be tested for drastic changes in content; neural networks can be used to detect whether the video clip contains important characters, objects, or scenes; optical flow algorithms can be used to determine whether there is intense motion in the video clip. These methods can be used to extract the key video clip from the video material.

[0064] Step 220: extract the audio feature vector of the initial video, and perform emotion feature detection on the audio feature vector; determine the basic audio segment in the audio feature vector based on the result of the emotion feature detection; detect the rhythm intensity and audio short-time energy in the basic audio segment; and determine the audio highlight segment based on the rhythm intensity detection result and the audio short-time energy detection result; determine at least one video highlight segment in the initial video based on the image highlight segment and the audio highlight segment.

[0065] Specifically, frame-level audio feature extraction is performed on the audio of the initial video. That is, only the audio is processed without processing the image, to obtain an audio feature vector of the initial video. Emotion detection is then performed on the audio feature vector. It is understood that in the art, emotion detection is a technique for inferring the speaker's emotional state through audio features (such as pitch, volume, speaking rate, etc.). Common emotion detection methods include, but are not limited to, traditional feature extraction based on support vector machines (SVMs) and feature extraction based on deep learning neural networks to determine the speaker's emotional state. This embodiment provides a specific emotion feature extraction method, which includes extracting frame-level audio features from the audio of the initial video using a HuBERT (Hidden-Unit BERT) model fine-tuned on a Chinese emotion dataset. Mutation detection is then performed using cosine distance to identify mutation points in the audio. The original audio is segmented according to the detected mutation points. Emotion recognition is then performed on the segmented audio using a trained emotion classifier to extract the aforementioned basic audio segments with drastic emotional changes. These emotions can include anger, happiness, surprise, and other emotions.

[0066] After detecting the basic audio segment, the rhythm intensity and audio short-time energy in the basic audio segment are detected. The audio can be converted into a spectrum, and the rhythm intensity of each frame of audio is calculated based on the spectral amplitude of each frame. After calculating the rhythm intensity of each frame, the audio rhythm mutation point in the basic audio segment can be determined based on the rhythm intensity. It can be understood that the time period where the audio rhythm mutation point is located is most likely the key segment, which can also be called the highlight segment. In addition, the short-time energy of each frame can be calculated. Short-time energy can be understood as the audio signal value. Based on the audio signal value, the change in audio volume can be reflected. Then, the peak value with higher short-time energy in the basic audio segment can be determined based on the audio short-time energy. Similarly, it can be understood that the segment with the short-time energy peak is most likely the highlight segment. In summary, the rhythm intensity monitoring results and the short-time energy detection results can be combined to determine the audio highlight segment. For example, the segment corresponding to each audio rhythm mutation point and the segment corresponding to the peak with higher short-time energy can be used as the above-mentioned audio highlight segment. The intersection between the segment corresponding to the audio rhythm mutation point and the segment corresponding to the peak with higher short-time energy can also be used as the above-mentioned audio highlight segment, and so on.

[0067] It can be understood that the above-mentioned video highlight clip is a video clip that includes both audio and corresponding images, so it is necessary to combine the video key clip with the audio highlight clip to obtain the video highlight clip. Specifically, based on the time corresponding to the video key clip, the corresponding video clip can be extracted from the initial video (the video clip is a clip cut out from the initial video, containing audio and corresponding video), and based on the time corresponding to the audio highlight clip, the corresponding video clip can be extracted from the initial video (similarly, the video clip is a clip cut out from the initial video, containing audio and corresponding video). The above-mentioned video highlight clip can be obtained by combining all video clips; similarly, the intersection between the video clip corresponding to the video key clip in the original video and the video clip corresponding to the audio highlight clip in the original video can also be calculated, and the video clip serving as the intersection can be determined as the above-mentioned video highlight clip.

[0068] Step S230 , based on the motion trajectory of the moving target in the video highlight segment, a target video highlight segment is screened out from the video highlight segment, and a video outline is generated based on the target video highlight segment.

[0069] Specifically, the above method generally obtains multiple video highlight clips. In order to ensure the readability of the final generated video outline, these video highlight clips need to be reorganized. Specifically, the motion trajectory of the moving target included in each video highlight clip is determined, and the clips including the moving target are filtered out from the video highlight clips to obtain the target video highlight clips, so that redundant pictures can be further deleted, and then the target video highlight clips are rearranged based on the time sequence to obtain the above video outline.

[0070] Through steps S210 to S230, a video outline of any initial video can be efficiently generated. On the basis of ensuring that the key segments in the initial video are extracted, the extracted materials can be rearranged according to the time sequence of the motion targets and video segments, which not only ensures the refinement and comprehensiveness of the information in the final generated video outline, but also ensures the readability of the video outline, allowing the audience to understand the content of the initial video more smoothly through the video outline.

[0071] In some embodiments, the rhythm intensity and audio short-time energy in the basic audio segment are detected, and the audio highlight segment is determined based on the rhythm intensity detection result and the audio short-time energy detection result, including:

[0072] Performing audio short-time energy detection on the basic audio clip, calculating the root mean square of the audio short-time energy detection results, and determining the volume of each frame of audio in the basic audio clip based on the root mean square calculation results;

[0073] Determine the rhythm intensity of each frame of audio according to the difference between the spectral amplitude of each frame of audio in the basic audio segment and the spectral amplitude of the historical frame;

[0074] Perform a fusion weighted calculation on the volume of each frame of audio and the rhythm intensity of each frame of audio to obtain a highlight score for each frame of audio in the basic audio clip;

[0075] An audio highlight segment is obtained based on all audio frames whose highlight scores are greater than a preset score threshold.

[0076] Specifically, audio short-term energy detection is performed on the basic audio clip, and the audio short-term energy (STE) is calculated as follows:

[0077] ;

[0078] Among them, E n Refers to the short-time energy of the nth frame, x(n) refers to the audio signal value, m refers to the local sampling point index within the frame, and N refers to the frame length. The root mean square (RMS) of the audio short-time energy detection result is then calculated to measure the volume change, as follows:

[0079] ;

[0080] Thus, the volume of the n-th frame audio can be calculated.

[0081] Furthermore, the basic audio clip is subjected to spectrum conversion processing. The rhythm intensity of each frame is determined based on the difference between the spectrum amplitude of each frame of audio after spectrum conversion and the spectrum amplitude of the historical frame (preferably the frame before the current frame). The calculation is as follows:

[0082] ;

[0083] Among them, O(n) refers to the rhythm intensity of the nth frame, S(n, k) refers to the spectral amplitude of the kth frequency, and ω(k) refers to the weight, which can be used to control the impact of changes in various frequency components (such as sounds in different frequency bands) on the rhythm intensity (or the perception of audio changes). The importance of different frequency bands in rhythm analysis can be adjusted according to actual needs. In order to treat all frequency points equally, the weight can be set to 1. At this time, it is assumed that all frequency components have equal contributions to the rhythm. Similarly, in some complex scenarios, ω(k) can be adjusted to more accurately capture rhythm changes or emotional fluctuations in the audio.

[0084] After calculating the volume and rhythm intensity of each frame of audio, for ease of calculation, the two can be normalized first. The calculation is as follows:

[0085] ;

[0086] ;

[0087] Where R(n) is the RMS energy of the nth frame, O(n) is the rhythm intensity of the nth frame, and is the normalized value in the range of [0,1]. Furthermore, the volume of each frame of audio and the rhythm intensity of each frame of audio are fused and weighted to obtain the highlight score of each frame of audio in the basic audio clip. The joint highlight score of each frame of audio is calculated as follows:

[0088] ;

[0089] Wherein, H(n) refers to the highlight score of the nth frame, and α∈[0,1] is a weight parameter, which indicates the balance between the importance of volume and rhythm. In practical applications, the value of α can be adjusted according to actual needs to adjust the importance of volume and rhythm accordingly. In some embodiments, α can be set to 0.5, that is, the importance of volume and rhythm is set to the same level.

[0090] Finally, the audio frames whose highlight scores are greater than a preset score threshold can be used as audio highlight segments. In some preferred embodiments, the highlight points can be obtained by using the threshold median method, which is calculated as follows:

[0091] ;

[0092] ;

[0093] Among them, T H It is the 90th percentile of all highlight score values, that is, the set highlight detection threshold. HighlightIndices represents the frame index of all audio frames with highlight detection scores greater than the threshold. Finally, the frame index is converted into a timestamp to obtain the corresponding audio highlight clip.

[0094] In some embodiments, determining a key video segment in an initial video includes:

[0095] Obtaining video key frames in the initial video, and dividing the video material in the initial video into multiple video segments based on the video key frames;

[0096] All video clips are traversed, and the key video clips are determined based on the video feature vectors of each video clip.

[0097] Specifically, this embodiment provides a method for obtaining key video segments, including first determining video key frames in an initial video. These video key frames typically include scene changes, thereby dividing the video material of the initial video into multiple video segments based on different scenes. In practical applications, ffmpeg (Fast Forward Moving Picture Experts Group) can be used to separate the video and audio materials in the original video material. Then, based on the scene changes, image key frames can be extracted from the video material. The calculation is as follows:

[0098] ;

[0099] Among them, I t Indicates the video key frame at time t when there is a large scene change.

[0100] Then, feature extraction is performed on each video clip, and the key video clips in the initial video are determined. Methods for determining the key video clips include, in some embodiments, scoring the features of different video clips to screen out highlight clips. Methods for calculating the scores of each video clip include: calculating the highlight score of each video clip, and determining the video clip with the highest highlight score as the key video clip. The highlight score is calculated as follows:

[0101]

[0102] Among them, highlight_score is the final score of each video clip, visual_score is the visual score of the video clip, which can be generated by relevant technical personnel or through a well-trained neural network to generate a visual score for each video clip. Similarly, the above-mentioned emotion_score is an emotional score, which can be generated by relevant technical personnel or through methods such as neural networks to generate an emotional score for the people in each video clip. Similarly, the above-mentioned object_score is a target score, which can be used to score the objects appearing in the video clip by relevant technical personnel or neural networks. In other embodiments, it can be determined whether there are drastic changes in the video clip based on the changes in the images between frames. If so, the video clip is determined to be a key video clip, and so on. Two methods for determining key video clips are given above, and relevant technical personnel can also design a method for determining key video clips according to actual needs.

[0103] In some embodiments, determining at least one video highlight segment in the initial video based on the video key segment and the audio highlight segment includes:

[0104] In the time dimension, based on the timestamps corresponding to the video key segments and the timestamps corresponding to the audio highlight segments, the initial video highlight segments are extracted from the initial video;

[0105] Fusing the image feature vector and the audio feature vector corresponding to the highlight segment of the initial video to obtain a fused feature vector of the highlight segment of the initial video;

[0106] Obtain the prompt words corresponding to the video outline and extract the text feature vector of the prompt words;

[0107] The text feature vector is matched with the fused feature vector for similarity to obtain the similarity corresponding to each fused feature vector, and the video highlight segment is determined based on the fused feature vector whose similarity is greater than a preset similarity threshold.

[0108] Specifically, based on the time corresponding to the video key segment and the time corresponding to the audio highlight segment, the initial video highlight segment is extracted from the initial video, wherein the initial video highlight segment is a segment extracted from the initial video based on the above time information, and includes audio and corresponding video.

[0109] Then, the image feature vector and audio feature vector of each frame in the highlight clip of the initial video are fused. In the time dimension, the extracted audio feature vector and image feature vector are fused to obtain the above-mentioned fused feature vector, that is, the audio feature vector and image feature vector at the same moment, and the fused feature vector at the corresponding moment is obtained. Specifically, the feature vectors corresponding to the audio and video are extracted respectively, that is, the above-mentioned image feature vector and audio feature vector. The calculation of the image feature vector and the audio feature vector is as follows:

[0110] ;

[0111] ;

[0112] Among them, the image feature vector F V The audio feature vector F can be extracted through the CLIP4Clip model. A It can be extracted through Mel spectrum. Furthermore, in order to ensure the consistency in subsequent calculations, the audio feature vector and the video frame feature vector need to be normalized to the time axis corresponding to the original video frame rate and aligned. Figure 3 As shown, the video frame feature vector and the audio feature vector are fused in the time dimension to obtain a fused feature vector. The final fused feature vector at time t can be recorded as: Ft=[F V ',F A '], it can be understood that the video frame feature vector is generally two-dimensional, and the audio feature vector is one-dimensional.

[0113] Then perform similarity matching between the text feature vector and the fusion feature vector to calculate the text feature vector F T and the cosine similarity between the fused feature vector Ft, wherein the text feature vector can be obtained based on the prompt word input by the user, such as in a football game, the user can input words such as "football" and "goal" as prompt words. Specifically, the text feature vector of the input prompt word can also be determined by the CLIP4Clip model. The above-mentioned similarity threshold can be set dynamically, for example, it can be set to the average score of the top n similarity scores, and n can be set manually, generally set to 3. The video clip corresponding to the fused feature vector with a similarity greater than the above-mentioned similarity threshold is recognized as a video highlight clip. The present application further matches the fused feature vector in the initial video highlight clip with the prompt word corresponding to the video outline, so that on the basis of extracting the key clips in the initial video, it can match again based on the prompt word input by the user, and obtain a video highlight clip that is more in line with the required video outline.

[0114] In some embodiments, extracting an initial video highlight segment from an initial video based on a timestamp corresponding to a video key segment and a timestamp corresponding to an audio highlight segment includes:

[0115] Restore the timestamps corresponding to the audio highlight clips to the timeline of the original video to obtain the audio highlight index corresponding to each audio highlight clip;

[0116] Restore the timestamps corresponding to the key segments of the video to the timeline of the original video to obtain the video key frame index corresponding to each key segment of the video;

[0117] At least one initial video highlight segment in the initial video is obtained based on the audio highlight index and the video key frame index.

[0118] Specifically, this embodiment provides a method for extracting corresponding video highlight segments after respectively calculating audio highlight segments and video key segments.

[0119] By converting the timestamps corresponding to the video key segments and the audio highlight segments into frame indexes, the corresponding segments in the original video or audio can be extracted based on these frame indexes and fused to obtain the above-mentioned video highlight segments. In some embodiments, the extracted audio highlight segments and the image highlight segments may not be exactly the same, so the union of the audio highlight segments and the image highlight segments can be taken as candidate highlight segments. For example, the extracted audio highlight segments are from the 2nd to the 5th seconds in the original audio, and the extracted video highlight segments are from the 3rd to the 6th seconds in the original video. At this time, the video segments from the 2nd to the 5th seconds can be extracted from the original video based on the audio highlight segments, and the video segments from the 3rd to the 6th seconds can be extracted from the original video based on the video highlight segments. The two video segments are combined, and the video segments from the 2nd to the 6th seconds are used as the above-mentioned video highlight segments.

[0120] In some embodiments, selecting a target video highlight segment from the video highlight segment based on a motion trajectory of a moving target in the video highlight segment includes:

[0121] Extract the motion trajectory of the moving target in each video highlight segment;

[0122] Performing anomaly detection on each motion trajectory, and determining a first motion trajectory among the motion trajectories based on the anomaly detection result; wherein the first motion trajectory includes a plurality of first sub-trajectories;

[0123] detecting a moving target in the first motion trajectory, and rearranging first sub-trajectories included in the first motion trajectory based on the moving target to obtain at least one second motion trajectory, wherein each second motion trajectory corresponds to a moving target;

[0124] Calculating a score for the second motion trajectory based on a dynamic programming algorithm, and determining a motion trajectory with a score greater than a preset second score threshold as a target motion trajectory; wherein the score calculation for the second motion trajectory is completed based on the video highlight score, the audio highlight score, and the motion trajectory highlight score of the second motion trajectory;

[0125] Based on the target motion trajectory, the corresponding target video highlight segment in the initial video is determined.

[0126] Specifically, after obtaining multiple video highlight segments, in order to improve the readability of the subsequently generated video outline and to further refine the extracted highlight videos, this embodiment selects target video highlight segments from the video highlight segments based on the motion trajectory of the moving target. Specifically, the above-mentioned video highlight segments usually contain a wealth of moving objects. First, the motion trajectory of the moving target in each video highlight segment is extracted. Existing neural networks, such as YOLO-v11, can be used to detect and track the moving target in each video highlight segment, and based on the tracking results, the motion trajectory of the moving target can be obtained. In actual applications, since the motion trajectory of the moving target may experience jitter, breakage, or even drift, in some preferred embodiments, the motion trajectory of the moving target can be optimized, and since the optimized motion trajectory may belong to multiple targets or different video segments, trajectory rearrangement and time rearrangement are required. Among them, trajectory optimization includes trajectory smoothing. When smoothing the trajectory, in order to reduce jitter, double exponential smoothing can be used, and the calculation is as follows:

[0127] ;

[0128] ;

[0129] in, and is the smoothed coordinate, x t with y t are the original coordinates, α is the smoothing factor, which can usually be set to 0.1≤α≤0.3.

[0130] After extracting the motion trajectory of the moving target corresponding to each video highlight segment, anomaly detection can be performed on each motion trajectory to obtain the first motion trajectory in each motion trajectory. It can be understood that in this embodiment, anomaly detection can find the moments of sudden and drastic changes in the motion of the moving target. These moments are often the key points of the highlight moment. If the motion anomaly is ignored, the wonderful video segment may be missed. Among them, the above anomaly detection can be performed by acceleration a t To filter out the abnormal first motion trajectory, the calculation is as follows:

[0131] ;

[0132] in, is the velocity at time t, v t-1 is the speed at time t-1, and △t is the time difference between time t-1 and time t. t If the value is greater than the preset anomaly detection threshold, it can be considered that the target has undergone a sudden movement (such as sudden stop, sprint, fall, etc.) at time t, and this period can be recorded as an abnormal trajectory highlight segment.

[0133] In actual applications, the segment corresponding to the above-mentioned first motion trajectory may correspond to multiple motion targets, and there are different first sub-trajectories corresponding to different motion targets. The relationship between the motion target and the first sub-trajectory is a one-to-many or one-to-one relationship, and the motion target corresponding to each motion trajectory is not clear. Therefore, in order to improve the quality of the subsequently generated video outline and to improve the logic of the video display, it is necessary to rearrange the first sub-trajectory based on the motion target, that is, to organize the sub-trajectories belonging to the same target into the same segment to obtain the second motion trajectory. At this time, there is a one-to-one correspondence between the second motion trajectory and the motion target.

[0134] To ensure the quality of the highlight video, the second motion trajectory can be scored using a dynamic programming algorithm. The motion trajectory with a score greater than a preset second score threshold is determined as the target motion trajectory. The trajectory score is calculated as follows:

[0135] ;

[0136] in, represents the highlight score of track i, △T i represents the duration of the video clip corresponding to track i, T max Indicates the maximum duration of the video outline, where the highlight score of track i It can be calculated by the following formula.

[0137] ;

[0138] in, represents the visual highlight score of the i-th track, represents the audio highlight score of the i-th track, represents the motion highlight score of the i-th track (which can be obtained by the above track anomaly detection), ω V 、ω A 、ω M is the weight coefficient, satisfying ω V +ω A +ω M =1, which can be set according to the relative importance of video, audio and motion.

[0139] Specifically, the motion highlight score is calculated as follows:

[0140] ;

[0141] Among them, a t represents the acceleration of the t-th frame, θ represents the set abnormal acceleration threshold, L represents the indicator function, which is 1 if the condition is met, otherwise it is 0, T i Indicates the total number of frames of the trajectory.

[0142] The visual highlight score is calculated as follows:

[0143] ;

[0144] Among them, c t Indicates the confidence of the detection box, △s t It represents the rate of change of the detection box area, and has:

[0145] ;

[0146] and d t It represents the distance between the normalized target center and the center of the picture. a1, a2, and a3 are weight coefficients, satisfying a1+a2+a3=1. Indicates the total number of frames of the trajectory.

[0147] The audio highlight score is calculated as follows:

[0148] ;

[0149] Among them, RMS t represents the root mean square of the short-time energy detection result of the audio at time t, O(t) represents the rhythm intensity at time t, It is a parameter that can adjust the importance of rhythm and is less than 1.

[0150] In summary, the above method can be used to further refine the extracted video highlight segments to obtain the target video highlight segments after trajectory rearrangement. Moreover, since the trajectories are rearranged in this embodiment, the readability of the finally generated video outline can be effectively improved.

[0151] In some embodiments, determining a corresponding target video highlight segment in an initial video based on the target motion trajectory includes:

[0152] determining a target sub-motion trajectory in the target motion trajectory, and rearranging the target sub-motion trajectory in chronological order to obtain a third motion trajectory corresponding to the target sub-motion trajectory;

[0153] The target video highlight segment corresponding to the initial video is obtained based on the third motion trajectory.

[0154] After obtaining the target motion trajectory, the target sub-motion trajectory included in the target motion trajectory can be determined. Each target sub-motion trajectory corresponds to a moving target. In order to improve the logic of subsequent video presentation, the target sub-motion trajectory can be rearranged in chronological order to obtain a third motion trajectory corresponding to the target sub-motion trajectory. Time rearrangement can make the sequence of highlight events more coherent, enhance the viewing experience, and reduce redundancy to avoid the repetition of the same scene. It can be understood that when the motion trajectory is time-rearranged, the essence is to rearrange the video clips corresponding to the motion trajectory in chronological order.

[0155] In some preferred embodiments, the target sub-motion trajectories are time-rearranged and sorted using an optimal path algorithm, specifically:

[0156] ;

[0157] in, Representation fragment and The smaller the similarity, the better. N is the number of segments. By optimizing the sorting, the transition between each target sub-motion track can be made smoother. Finally, after obtaining the third motion track, the highlight segment of the target video is obtained based on the segment corresponding to the third motion track in the initial video.

[0158] In summary, the target video highlight segments corresponding to the initial video can be obtained, and the above-mentioned video outline can be obtained by integrating the target video highlight segments.

[0159] This application also provides a preferred embodiment for generating a video outline. Figure 4 This is a video outline generation method in a preferred embodiment.

[0160] Step S410: obtaining the original video and performing preprocessing on the original video, such as noise reduction and feature extraction.

[0161] Step S420, performing highlight detection on the original video, that is, performing highlight detection on the audio of the original video respectively to obtain audio highlight segments, and similarly performing highlight detection on the video frame material of the original video to obtain the above-mentioned video key segments, and finally obtaining multiple video highlight segments in combination.

[0162] Step S430 : detecting and tracking a moving target in the highlight segment of the video to obtain a first motion trajectory, wherein the first motion trajectory includes a plurality of first sub-trajectories.

[0163] Step S440 involves performing trajectory optimization and temporal reordering on the first sub-trajectory. This involves first reordering the first sub-trajectory based on the target, generating multiple second trajectories that correspond one-to-one with the target. Highlight scores are then calculated for the second trajectories, with those with scores exceeding a second score threshold being designated as the target trajectory. The sub-trajectories within the target trajectory are then chronologically reordered to generate a third trajectory, the final target trajectory. This improves the readability of the subsequently generated video outline and makes the video more logical. This step ensures the quality of the final trajectory through temporal reordering, trajectory reordering, and score calculation of multiple trajectories.

[0164] Step S450: Obtain the target video highlight segment through the final motion trajectory, and then obtain the video outline corresponding to the original video.

[0165] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0166] Based on the same inventive concept, embodiments of the present application also provide a video outline generation device for implementing the aforementioned video outline generation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more video outline generation device embodiments provided below can be found in the aforementioned limitations of the video outline generation method and will not be further elaborated here.

[0167] In one embodiment, Figure 5 As shown, a video outline generation device is provided, including: an acquisition module 51, a calculation module 52 and a generation module 53, wherein:

[0168] An acquisition module 51 is configured to acquire an initial video and determine key video segments in the initial video;

[0169] A calculation module 52 is configured to extract an audio feature vector of the initial video, perform emotion feature detection on the audio feature vector, determine a basic audio segment in the audio feature vector based on the result of the emotion feature detection, detect rhythm intensity and audio short-time energy in the basic audio segment, and determine an audio highlight segment based on the rhythm intensity detection result and the audio short-time energy detection result; and determine at least one video highlight segment in the initial video based on the video key segment and the audio highlight segment.

[0170] The generating module 53 is configured to filter out a target video highlight segment from the video highlight segments based on the motion trajectory of the moving target in the video highlight segment, and generate the video outline based on the target video highlight segment.

[0171] Each module in the video outline generation device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0172] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to video outline generation. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a video outline generation method is implemented.

[0173] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0174] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0175] Obtaining an initial video and determining a key video segment in the initial video;

[0176] Extracting an audio feature vector from the initial video and performing emotional feature detection on the audio feature vector; determining a basic audio segment in the audio feature vector based on the results of the emotional feature detection; detecting rhythm intensity and audio short-time energy in the basic audio segment; and jointly determining an audio highlight segment based on the rhythm intensity detection results and the audio short-time energy detection results; and determining at least one video highlight segment in the initial video based on the video key segment and the audio highlight segment;

[0177] Based on the motion trajectory of the moving target in the video highlight clips, the target video highlight clips are screened out from the video highlight clips, and the video outline is generated based on the target video highlight clips.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0179] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0180] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for generating a video outline, characterized in that: The method comprises: Obtaining an initial video and determining a key video segment in the initial video; Extracting an audio feature vector of the initial video, performing emotional feature detection on the audio feature vector, determining a basic audio segment in the audio feature vector based on the result of the emotional feature detection, detecting rhythm intensity and audio short-time energy in the basic audio segment, and jointly determining an audio highlight segment based on the rhythm intensity detection result and the audio short-time energy detection result; determining at least one video highlight segment in the initial video based on the video key segment and the audio highlight segment; Based on the motion trajectory of the moving target in the video highlight segment, target video highlight segments are screened out from the video highlight segment, and the video outline is generated based on the target video highlight segment, wherein the step of screening out target video highlight segments from the video highlight segment based on the motion trajectory of the moving target in the video highlight segment comprises: extracting the motion trajectory of the moving target in each of the video highlight segments; Performing anomaly detection on each of the motion trajectories, and determining a first motion trajectory among the motion trajectories based on the anomaly detection result; wherein the first motion trajectory includes a plurality of first sub-trajectories; detecting a moving target in the first motion trajectory, and rearranging the first sub-trajectories included in the first motion trajectory based on the moving target to obtain at least one second motion trajectory, wherein each second motion trajectory corresponds to one of the moving targets; Calculating a score for the video clip corresponding to the second motion trajectory based on a dynamic programming algorithm, and determining a motion trajectory having a score greater than a preset second score threshold as a target motion trajectory; wherein the score calculation for the second motion trajectory is completed based on the video highlight score, the audio highlight score, and the motion trajectory highlight score of the second motion trajectory; Based on the target motion trajectory, a corresponding target video highlight segment in the initial video is determined.

2. The method according to claim 1, characterized in that The detecting of the rhythm intensity and the audio short-time energy in the basic audio segment and determining the audio highlight segment based on the rhythm intensity detection result and the audio short-time energy detection result includes: Performing audio short-time energy detection on the basic audio segment, calculating a root mean square of the audio short-time energy detection results, and determining the volume of each frame of audio in the basic audio segment based on the root mean square calculation results; determining the rhythm intensity of each frame of audio according to a difference between a spectrum amplitude of each frame of audio in the basic audio segment and a spectrum amplitude of a historical frame; Performing a fusion weighted calculation on the volume of each frame of audio and the rhythm intensity of each frame of audio to obtain a highlight score for each frame of audio in the basic audio segment; The audio highlight segment is obtained based on all audio frames whose highlight scores are greater than a preset score threshold.

3. The method according to claim 1, characterized in that The determining of the key video segments in the initial video includes: Obtaining video key frames in the initial video, and dividing the video material in the initial video into a plurality of video segments based on the video key frames; All the video segments are traversed, and the key video segments are determined based on the video feature vectors of the respective video segments.

4. The method according to claim 1, wherein The determining, based on the video key segment and the audio highlight segment, at least one video highlight segment in the initial video includes: In the time dimension, based on the timestamp corresponding to the video key segment and the timestamp corresponding to the audio highlight segment, extracting the initial video highlight segment from the initial video; Fusing the image feature vector and the audio feature vector corresponding to the highlight segment of the initial video to obtain a fused feature vector of the highlight segment of the initial video; Obtaining a prompt word corresponding to the video outline and extracting a text feature vector of the prompt word; The text feature vector is matched with the fused feature vector for similarity to obtain the similarity corresponding to each fused feature vector, and the video highlight segment is determined based on the fused feature vector whose similarity is greater than a preset similarity threshold.

5. The method according to claim 4, characterized in that The extracting the initial video highlight segment from the initial video based on the timestamp corresponding to the video key segment and the timestamp corresponding to the audio highlight segment includes: Restoring the timestamps corresponding to the audio highlight segments to the timeline of the original video to obtain the audio highlight index corresponding to each audio highlight segment; Restoring the timestamps corresponding to the key segments of the video to the timeline of the original video to obtain the video key frame index corresponding to each key segment of the video; At least one initial video highlight segment in the initial video is obtained based on the audio highlight index and the video key frame index.

6. The method according to claim 1, characterized in that The determining, based on the target motion trajectory, the target video highlight segment corresponding to the initial video includes: determining target sub-motion trajectories in the target motion trajectory, and rearranging all the target sub-motion trajectories in chronological order to obtain a third motion trajectory corresponding to all the target sub-motion trajectories; The target video highlight segment corresponding to the initial video is obtained based on the third motion trajectory.

7. A video outline generating device, characterized in that: The device comprises: An acquisition module, configured to acquire an initial video and determine a key video segment in the initial video; a calculation module, configured to extract an audio feature vector of the initial video, perform emotion feature detection on the audio feature vector, determine a basic audio segment in the audio feature vector based on the result of the emotion feature detection, detect rhythm intensity and audio short-time energy in the basic audio segment, and jointly determine an audio highlight segment based on the rhythm intensity detection result and the audio short-time energy detection result; and determine at least one video highlight segment in the initial video based on the video key segment and the audio highlight segment; A generating module is configured to filter out target video highlight segments from the video highlight segments based on the motion trajectory of the motion target in the video highlight segments, and generate the video outline based on the target video highlight segments, wherein filtering out target video highlight segments from the video highlight segments based on the motion trajectory of the motion target in the video highlight segments comprises: extracting the motion trajectory of the motion target in each of the video highlight segments; Performing anomaly detection on each of the motion trajectories, and determining a first motion trajectory among the motion trajectories based on the anomaly detection result; wherein the first motion trajectory includes a plurality of first sub-trajectories; detecting a moving target in the first motion trajectory, and rearranging the first sub-trajectories included in the first motion trajectory based on the moving target to obtain at least one second motion trajectory, wherein each second motion trajectory corresponds to one of the moving targets; Calculating a score for the video clip corresponding to the second motion trajectory based on a dynamic programming algorithm, and determining a motion trajectory having a score greater than a preset second score threshold as a target motion trajectory; wherein the score calculation for the second motion trajectory is completed based on the video highlight score, the audio highlight score, and the motion trajectory highlight score of the second motion trajectory; Based on the target motion trajectory, a corresponding target video highlight segment in the initial video is determined.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images