Dynamic cover generation method and device, electronic equipment and storage medium

Abstract

The invention provides a dynamic cover generation method and device, electronic equipment and a storage medium, and relates to the technical field of live broadcast processing. According to the method, multi-modal data in a video stream is acquired, a continuous cover value score time sequence is generated based on the multi-modal data, then a core moment of a highlight event is accurately positioned through peak detection, a complete highlight event segment is intelligently expanded on the basis, and a dynamic cover is generated by using the highlight event segment. Therefore, the defects that the wonderful degree is difficult to quantify, false highlight interference is likely to happen, and fragment narration is incomplete in a traditional method are overcome, and end-to-end automation from instantaneous highlight recognition to high-quality dynamic cover generation is achieved. And the visual expressive force and the user attraction of the cover content and the real-time performance and the stability of the system in a real live broadcast scene are improved.

Description

Technical Field

[0001] This invention relates to the field of live streaming processing technology, and more specifically, to a method, apparatus, electronic device, and storage medium for generating dynamic cover images. Background Technology

[0002] With the rapid development of online live streaming platforms, live streaming content has exploded, making it a major challenge for users to quickly identify and select content of interest from massive video streams. Traditional static covers are usually manually captured by the host or generated by the system by randomly selecting keyframes. They often fail to accurately reflect the core highlights of the live streaming content and lack the ability to guide the audience's attention, resulting in unattractive covers and low click-through rates. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, apparatus, electronic device and storage medium for generating dynamic covers.

[0004] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, the present invention provides a method for generating a dynamic cover, the method comprising: Acquire multimodal data from the video stream; Based on the multimodal data, a time series of cover value scores is obtained; Peak detection is performed on the time series of the cover value to determine at least one highlight peak time point; Based on each of the aforementioned peak time points, a highlight event segment is obtained; A dynamic cover is generated based on the aforementioned highlight event fragment.

[0005] Optionally, the step of obtaining the cover value time series based on the multimodal data includes: The multimodal data is input into a pre-trained cover value scoring model for processing to obtain the cover value score time series.

[0006] Optionally, the training process of the cover value scoring model includes: Construct multiple comparison sample pairs, wherein each comparison sample pair includes a first sample and a second sample, wherein the first sample represents a video clip that is more suitable as a cover than the second sample; Each of the comparison samples is used to train the input preset model, and the training ends when the difference between the score of the first sample and the score of the second sample output by the preset model is greater than a preset boundary value. The resulting trained preset model is used as the cover value scoring model.

[0007] Optionally, the step of constructing multiple comparison sample pairs includes: For any two candidate video segments in the video stream, identify the content implementation difficulty in each candidate video segment, and use the candidate video segment with higher content implementation difficulty as the first sample, and the candidate video segment with lower content implementation difficulty as the second sample; or, The bullet comments in each candidate video segment are counted. The candidate video segments that elicit a higher number of positive emotional bullet comments are selected as the first sample, and the candidate video segments that elicit a lower number of positive emotional bullet comments are selected as the second sample; or, For each candidate video segment, the human pose is estimated, and the candidate video segments with larger motion amplitudes are used as the first sample, and the candidate video segments with smaller motion amplitudes are used as the second sample; or, The system receives a user's selection operation for two candidate video segments, uses the selected candidate video segment as the first sample, and uses the candidate video segment that was not selected by the user as the second sample.

[0008] Optionally, the multimodal data includes the original video frame sequence, the original audio signal, and the real-time bullet screen stream, and the cover value scoring model includes a three-dimensional convolutional neural network, an audio encoder, a natural language processing module, and an evaluation module; The step of inputting the multimodal data into a pre-trained cover value scoring model to obtain a time series of cover value scores includes: The three-dimensional convolutional neural network is used to extract the pure visual features of the original video frame sequence; The acoustic features of the original audio signal are extracted using the audio encoder. The semantic features of the real-time bullet screen stream are extracted using the natural language processing module. The acoustic features and the semantic features are fused to obtain multimodal contextual features; The pure visual features and the multimodal contextual features are input into the evaluation module to obtain the cover value score time series.

[0009] Optionally, the step of performing peak detection on the time series of the cover value to determine at least one highlight peak time point includes: Each local maximum point in the time series of the cover value is identified as a candidate peak. The candidate peaks are filtered and corrected by combining the semantic information obtained from the video stream, and the time point corresponding to each candidate peak that passes the correction is determined as the highlight peak time point.

[0010] Optionally, the step of obtaining a highlight event segment based on each of the highlight peak time points includes: Taking each of the aforementioned peak time points as the center, the process extends forward and backward along the time axis of the video stream until a preset stopping condition is met, thereby obtaining the start time point and end time point of the highlight event. The highlight event segment is obtained from the video stream based on the event start time and the event end time.

[0011] Optionally, the step of generating a dynamic cover based on the highlight event fragment includes: The highlight event segments are subjected to duration normalization to obtain candidate segments; Obtain the peak score, average score, and duration features of the candidate segments; The overall quality score of the candidate segment is obtained based on the peak score, the average score, and the duration feature. If the overall quality score is not less than a preset threshold, a dynamic cover is generated based on the candidate segments.

[0012] Secondly, the present invention provides a dynamic cover generation device, the device comprising: The acquisition module is used to acquire multimodal data from the video stream; The processing module is used to obtain a cover value time series based on the multimodal data; perform peak detection on the cover value time series to determine at least one highlight peak time point; obtain highlight event segments based on each highlight peak time point; and generate a dynamic cover based on the highlight event segments.

[0013] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to implement the dynamic cover generation method described in the first aspect above.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the dynamic cover generation method as described in the first aspect above.

[0015] The dynamic cover generation method, apparatus, electronic device, and storage medium provided in this invention involve acquiring multimodal data from a video stream; obtaining a cover value time series based on the multimodal data; performing peak detection on the cover value time series to determine at least one highlight peak time point; obtaining highlight event segments based on each highlight peak time point; and generating a dynamic cover based on the highlight event segments. Because this invention generates a continuous cover value time series by acquiring multimodal data from a video stream, and then accurately locates the core moment of the highlight event through peak detection, and intelligently expands it to a complete highlight event segment; and uses the highlight event segment to generate a dynamic cover, it overcomes the shortcomings of traditional methods such as difficulty in quantifying "excitement level," susceptibility to false highlight interference, and incomplete segment narrative. It achieves end-to-end automation from instantaneous highlight recognition to high-quality dynamic cover generation, improving the visual expressiveness of the cover content, user appeal, and the real-time performance and stability of the system in real live streaming scenarios.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This figure shows a schematic block diagram of an electronic device provided by an embodiment of the present invention; Figure 2 A flowchart illustrating a dynamic cover generation method provided by an embodiment of the present invention is shown; Figure 3 The diagram illustrates a flowchart of a cover value scoring model training method provided by an embodiment of the present invention. Figure 4 The diagram shows a functional block diagram of a dynamic cover generation device provided in an embodiment of the present invention.

[0019] Icons: 100 - Electronic device; 110 - Memory; 120 - Processor; 130 - Communication module; 200 - Dynamic cover generation device; 201 - Acquisition module; 202 - Processing module. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0022] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0023] With the rapid development of the online live streaming industry, especially the increasing popularity of real-time content formats such as games, sports, and entertainment, how to efficiently and accurately present the core highlights of live streaming content has become a key issue in improving content distribution efficiency and user engagement. Dynamic covers, as a multimedia summary format capable of displaying the core visual information of a video, have been widely used in scenarios such as live stream replays and short video recommendations. An ideal dynamic cover should be short in duration (usually 5-10 seconds), have a complete narrative, a tight pace, and accurately convey the "most exciting" moment.

[0024] However, automatically generating high-quality animated covers in long, continuous video streams still faces many technical challenges. Existing related technologies mainly focus on the following areas, but all have significant limitations: First, some solutions employ absolute scoring mechanisms based on classification or regression models to evaluate the "highlighting" of video clips. These methods typically rely on manually labeled tags (e.g., highlight / non-highlight) or pre-defined score mappings, attempting to predict the absolute value score of each clip through supervised learning. However, "highlighting" is a highly subjective, continuous, and context-dependent concept, difficult to accurately characterize with limited discrete labels. Furthermore, these methods cannot effectively distinguish subtle differences between different levels of highlight events (e.g., the quality difference between a "pentakill" and a "double kill"), resulting in a lack of learning ability regarding relative merits, thus affecting the final quality ranking and selection of the cover.

[0025] Secondly, some technologies attempt to incorporate multimodal information fusion (such as audio, bullet comments, and speech recognition) to enhance judgment accuracy. Although these non-visual signals reflect audience emotions or broadcaster reactions to some extent, existing methods often treat each modality equally or simply splice features together, failing to establish a reasonable primary-secondary structure. This makes the system susceptible to "pseudo-highlights"—for example, a broadcaster shouting due to emotional excitement without significant visual change may be misjudged as a highlight moment; or visually similar "wonderful kill" and "awkward death" may be incorrectly assigned due to audio / bullet comment differences, thus generating misleading cover images.

[0026] Furthermore, most existing technologies focus on instantaneous highlight detection, that is, identifying the "climax moment" in a frame or a very short period of time, but lack a systematic expansion mechanism from "moment" to "usable fragments". Most solutions generate candidate fragments by using fixed-duration cropping or heuristic fore-and-aft extraction, ignoring the integrity and temporal logic of the event itself. This approach easily leads to problems such as abrupt start and end of fragments, narrative breaks, and unbalanced rhythm, seriously affecting user experience and cover appeal.

[0027] Finally, from a system architecture perspective, current implementations generally couple analysis and generation within the same processing flow, lacking a comprehensive design that considers real-time performance and stability. In high-concurrency, low-latency live streaming environments, if complex model inference and segment generation tasks are executed synchronously, performance bottlenecks can easily occur, leading to delayed responses or even service interruptions.

[0028] In summary, existing technologies have shortcomings in terms of modeling the "excitement level", multimodal information fusion strategies, construction logic of highlight segments, and system-level engineering implementation, making it difficult to meet the actual needs of automatically, in real time, and with high quality generating dynamic covers in real live streaming scenarios.

[0029] Therefore, embodiments of the present invention provide a dynamic cover generation method, apparatus, electronic device, and storage medium, which will be described in detail below.

[0030] Please refer to Figure 1 This is a block diagram of electronic device 100. Electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, processor 120, and communication module 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0031] The memory 110 is used to store programs or data. The memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0032] The processor 120 is used to read / write data or programs stored in the memory 110 and to perform corresponding functions.

[0033] The communication module 130 is used to establish a communication connection between the electronic device 100 and other communication terminals via a network, and to send and receive data via the network.

[0034] It should be understood that, Figure 1 The structure shown is only a schematic diagram of the electronic device 100. The electronic device 100 may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0035] Please refer to Figure 2 The dynamic cover generation method provided in this embodiment of the invention includes steps S101 to S105.

[0036] S101, acquire multimodal data from the video stream.

[0037] In this embodiment of the invention, multiple types of data can be synchronously acquired from a real-time video stream to form a multimodal input signal.

[0038] Multimodal data includes at least the original video frame sequence, the original audio signal, and the real-time bullet screen stream.

[0039] The original video frame sequence is a collection of image frames arranged in chronological order, used to capture changes in visual content.

[0040] The original audio signal contains sound information such as the broadcaster's voice, ambient sound effects, and background music.

[0041] Real-time bullet comments record text comments sent by viewers at specific times, reflecting users' immediate emotional reactions and the level of interaction.

[0042] These data, after being aligned with a unified timestamp, serve as the basic input for subsequent model processing.

[0043] S102, based on multimodal data, obtain the cover value score time series.

[0044] In a possible implementation, multimodal data can be input into a pre-trained cover value scoring model to obtain a cover value score time series.

[0045] This invention utilizes a deep neural network model trained with a sorting learning paradigm, namely the "cover value scoring model," to evaluate the relative value of the content at each moment in the video stream and output a continuous "cover value score time series."

[0046] Each value in the cover value score time series represents the potential attractiveness score of the video clip as a dynamic cover within the corresponding time window.

[0047] In this embodiment of the invention, the training process of the cover value scoring model may include, for example: Figure 3 S201~S202 are shown.

[0048] S201, construct multiple comparison sample pairs.

[0049] Each comparison sample pair includes a first sample and a second sample, where the first sample represents a video clip that is more suitable as a cover than the second sample.

[0050] In other words, each comparison sample pair consists of two video clips, namely the first sample (Clip_better) and the second sample (Clip_worse), with the first sample being more suitable as a dynamic cover than the second sample.

[0051] The superiority / inferiority relationship between the first and second samples can be established through automated rules or manual annotation. The methods include, but are not limited to, the following four: Method 1: For any two candidate video segments in the video stream, identify the content implementation difficulty of each candidate video segment, and use the candidate video segment with higher content implementation difficulty as the first sample and the candidate video segment with lower content implementation difficulty as the second sample.

[0052] Understandably, for any two candidate video clips in a video stream, optical character recognition (OCR) technology is used to extract the difficulty level of the game content displayed on the screen (e.g., "pentakill," "quadrakill," "triple kill," etc.). The clip with the higher difficulty level is designated as the first sample, and the one with the lower difficulty level as the second sample. For example, the "pentakill" clip is labeled as the first sample, and the "double kill" clip is labeled as the second sample.

[0053] Method 2: For any two candidate video segments in the video stream, count the bullet comments in each candidate video segment, and take the candidate video segment that triggers a higher number of positive emotional bullet comments as the first sample, and take the candidate video segment that triggers a lower number of positive emotional bullet comments as the second sample.

[0054] Understandably, by statistically analyzing the number of bullet comments and their emotional polarity during the playback of each candidate segment, natural language processing technology is used to identify positive emotional keywords (such as "666", "awesome", "wonderful", etc.), and the segment that triggers a higher frequency of positive bullet comments is taken as the first sample, and the other is taken as the second sample. For example, if a kill segment triggers 1200 "666" bullet comments, which is much higher than the 80 bullet comments of another segment, then the former is considered the first sample.

[0055] Method 3: For any two candidate video segments in the video stream, estimate the human pose in each candidate video segment, and use the candidate video segment with a larger motion range as the first sample and the candidate video segment with a smaller motion range as the second sample.

[0056] Understandably, posture estimation is performed on the characters' movements in the two candidate clips, and the energy or amplitude of change in the motion trajectory of key joints is calculated. Clips with more intense and larger amplitude movements (such as jumping to celebrate or performing vigorous operations) are used as the first sample, and clips with gentler movements are used as the second sample, thus reflecting the difference in visual impact.

[0057] Method 4: For any two candidate video segments in the video stream, receive the user's selection operation for the two candidate video segments, use the selected candidate video segment as the first sample, and use the candidate video segment that was not selected by the user as the second sample.

[0058] Understandably, by constructing a "choose one of two" annotation platform, annotators are presented with pairs of candidate video clips, and they only need to click "Left is better" or "Right is better" to complete the judgment. This semi-automated method can efficiently generate high-confidence comparison sample pairs, and is especially suitable for complex scenarios that are difficult to quantify through rules.

[0059] S202, train the input preset model with each comparison sample, and use the ranking loss function to adjust the model parameters of the preset model during the training process, so that the difference between the score of the first sample and the score of the second sample output by the model is greater than the preset boundary value, thus obtaining the cover value scoring model.

[0060] The constructed comparison samples are input into the deep learning model to be trained, and end-to-end training is performed using a ranking loss function (such as MarginRanking Loss). Let the model's output scores for the first and second samples be Score_better and Score_worse, respectively, then the loss function is defined as: Loss = max(0, margin - (Score_better - Score_worse)) Here, margin is a preset boundary value (e.g., 1.0) used to ensure that the model not only distinguishes between good and bad, but also maintains a certain score interval. By continuously adjusting the model parameters through backpropagation, a cover value scoring model that can accurately determine the relative quality of segments is finally obtained.

[0061] Understandably, this embodiment of the invention constructs multi-source heterogeneous comparison sample pairs (covering four types of methods: content implementation difficulty, bullet screen emotional intensity, human posture and movement amplitude, and manual "choose one of two" labeling), and uses a ranking loss function for end-to-end training. This enables the cover value scoring model to no longer rely on subjective, sparse, and difficult-to-label absolute score labels, but instead focus on learning fine-grained and verifiable relative quality relationships among massive fragments. This allows for accurate differentiation of visual events with similar semantics but vastly different values, effective suppression of "pseudo-highlight" interference, reduction of high-quality labeling costs, and enhancement of the breadth and robustness of training data coverage.

[0062] Furthermore, in this embodiment of the invention, the cover value scoring model adopts a visual-centric, multimodal-assisted neural network architecture, including a three-dimensional convolutional neural network (3D-CNN), an audio encoder, a natural language processing module, and an evaluation module. The implementation process of step S102 may include sub-steps S102-1 to S102-5.

[0063] S102-1 uses a three-dimensional convolutional neural network to extract pure visual features from the original video frame sequence.

[0064] In this embodiment of the invention, 3D-CNN can be used to extract dynamic visual features from video frame sequences simultaneously along the time and space dimensions, capturing key information such as action intensity, scene rhythm and visual coherence, forming "pure visual features".

[0065] S102-2, using an audio encoder to extract the acoustic features of the original audio signal.

[0066] In this embodiment of the invention, the audio encoder may employ a convolutional neural network or a Transformer structure to extract acoustic features such as intonation fluctuations, volume changes, applause / cheering from the audio waveform, reflecting emotional fluctuations at the auditory level.

[0067] S102-3 uses a natural language processing module to extract semantic features of the real-time bullet screen stream.

[0068] In this embodiment of the invention, a pre-trained language model can be used to encode the bullet screen text, extract its sentiment tendency, keyword density and contextual semantics, and generate a bullet screen semantic embedding vector.

[0069] S102-4 fuses acoustic and semantic features to obtain multimodal contextual features.

[0070] In this embodiment of the invention, audio features and bullet screen semantic features can be fused through an attention mechanism or feature splicing method to generate "multimodal context features" that reflect the collaborative influence of non-visual modalities, which can be used to supplement the contextual understanding of visual information.

[0071] S102-5, input pure visual features and multimodal contextual features into the evaluation module to obtain the cover value score time series.

[0072] In this embodiment of the invention, the evaluation module can be a fully connected network or a multi-head attention structure. By receiving dual inputs from the visual backbone and the multimodal branch, while maintaining visual dominance, it fuses contextual information and finally outputs the cover value score corresponding to each time window, forming a continuous scoring curve covering the entire video stream.

[0073] Understandably, this invention employs a "visual-centric, multimodal-assisted" neural network architecture. While ensuring visual content remains the core basis for judging cover value, it uses 3D-CNN to accurately model the essential features of dynamic images (such as action rhythm, compositional tension, and scene coherence). This is combined with emotional acoustic cues extracted from the audio encoder (such as cheering intensity and tone of voice) and semantic signals from the bullet comments parsed by the natural language processing module (such as emotional polarity and event reference). An attention-driven contextual fusion mechanism generates multimodal contextual features. Finally, the evaluation module achieves a collaborative decision-making process "primarily visual, supplemented by audio and text," thereby enhancing the model's ability to discern the true value of visual events. It can identify high-value pure visual highlights without strong audio, while eliminating false highlight interference, thus outputting a continuous cover value time series that conforms to human perception consistency. This lays the foundation for subsequent accurate location of highlight event segments.

[0074] S103, perform peak detection on the time series of the cover value to determine at least one highlight peak time point.

[0075] In a possible implementation, the process of implementing step S103 may include sub-steps S103-1 to S103-2.

[0076] S103-1 identifies each local maximum point in the cover value time series as a candidate peak.

[0077] In this embodiment of the invention, a peak detection algorithm can be applied to the scoring curve to detect all time points that satisfy the local maximum condition, which are then used as an initial candidate peak set.

[0078] S103-2, combining semantic information obtained from the video stream to filter and correct candidate peaks, and determining the time point corresponding to each corrected candidate peak as the highlight peak time point.

[0079] In this embodiment of the invention, false detections caused by “pseudo-highlights” can be eliminated by introducing contextual semantic analysis (such as bullet screen topic clustering and voice keyword recognition).

[0080] For example, if a high-scoring segment has a strong audio response but the video is still and no substantial event occurs (such as the anchor reading an announcement), it will be removed from the candidate set.

[0081] For example, when multiple adjacent peaks belong to the same continuous event, the highest score among them is selected as the representative highlight peak time point t_peak.

[0082] Understandably, this invention overcomes the limitations of traditional pure numerical threshold or mathematical extreme value detection by introducing contextual semantic analysis in the peak detection stage. By combining real-time semantic information, it verifies the contextual consistency and merges redundancies of the initial candidate peaks. This not only accurately eliminates false high-scoring points caused by "pseudo-highlights" and ensures that each highlight peak corresponds to a real visually dominant event, but also merges adjacent homogeneous peaks and retains the most representative core time point t_peak. This improves the semantic accuracy, event integrity, and physical interpretability of highlight localization, providing support for the subsequent construction of high-quality dynamic cover segments with coherent narratives, tight rhythms, and value focus.

[0083] S104, based on each highlight peak time point, obtains highlight event fragments.

[0084] In a possible implementation, the process of implementing step S104 may include sub-steps S104-1 to S104-2.

[0085] S104-1, taking each highlight peak time point as the center, extends forward and backward along the time axis of the video stream until the preset stopping condition is met, to obtain the start time point and end time point of the highlight event.

[0086] During the expansion process, two termination conditions are continuously monitored: (1) Scene switching is detected, that is, sudden changes in the content of the screen are identified through inter-frame differences or keyframe analysis; (2) If the cover value score is found to be consistently below a certain preset low threshold (such as 30% of the average), it indicates that the core event area has been left.

[0087] When any condition is met in either the forward or backward direction, the expansion in that direction stops.

[0088] S104-2, obtain highlight event segments from the video stream based on the event start time and event end time.

[0089] In this embodiment of the invention, based on the determined start and end times of the event, corresponding video, audio, and bullet screen data can be extracted from the original video stream to form a complete highlight event segment.

[0090] Understandably, this invention defines highlight event segments by using the peak time point as an anchor point, bidirectional adaptive expansion, and combining "scene mutation detection" and "value decay threshold" as dual termination conditions. This achieves a leap from isolated "highlight moments" to semantically complete, structurally reasonable, and duration-appropriate highlight event segments. This avoids narrative breaks caused by fixed window truncation and overcomes the problem of blurred boundaries due to noise interference when relying solely on the scoring curve. At the same time, visual continuity is ensured by identifying content switching through inter-frame differences, and the event's end is determined by the continuous decline in value score. This gives the obtained highlight event segments an inherent sense of rhythm and event closure, providing a material basis for generating dynamic covers with high click-through rates and strong dissemination.

[0091] S105 generates a dynamic cover based on highlight event fragments.

[0092] In a possible implementation, the process of implementing step S105 may include sub-steps S105-1 to S105-4.

[0093] S105-1, the duration of the highlight event segment is standardized to obtain candidate segments.

[0094] Since the natural duration of different events varies, they need to be standardized to a suitable display length (e.g., 5-10 seconds). Therefore, highlight event clips are cropped or time-scaled (e.g., variable speed playback to maintain the integrity of key actions) to generate standardized candidate clips.

[0095] S105-2, obtain the peak score, average score and duration features of the candidate segments.

[0096] Among them, the peak score, Score_peak=max(S(t)), is used to reflect the quality of the most exciting moment.

[0097] The average score, Score_avg = mean(S(t)), is used to measure the overall performance level. The normalized duration score, Score_duration = normalize(T), is calculated by setting an "ideal duration" (e.g., 8 seconds). The closer the actual duration of a candidate segment is to the set ideal duration, the higher the normalized duration score. For example, Score_duration = 1 - abs(T - 8) / 8. (S105-3) S105-3, based on peak score, average score and duration characteristics, the overall quality score of the candidate segment is obtained.

[0098] Calculate the overall quality score of the candidate segment using the following formula: Holistic_Score=w_peak Score_peak+w_avg Score_avg+w_duration Score_duration In the formula, the weights w_peak, w_avg, and w_duration should be set to maximize w_peak to highlight the importance of the climax. For example, w_peak=0.6, w_avg=0.3, and w_duration=0.1.

[0099] S105-4, If the overall quality score is not less than a preset threshold, then generate a dynamic cover based on the candidate segments.

[0100] When the overall quality score of the candidate segments is not less than the preset threshold, the selected candidate segments are packaged into a dynamic cover in a standard format (such as MP4 or GIF) by calling an asynchronous rendering task, and then pushed to the content management system or recommendation platform to complete the closed loop of the entire process.

[0101] Understandably, the embodiments of the present invention take into account both visual integrity and platform specifications through duration standardization, and construct a multi-dimensional quality assessment method that takes into account "explosiveness" (peak score), "stability" (average score) and "adaptability" (duration score). The overall quality score is generated by a weighted fusion method that highlights the dominance of peak values. Finally, by making a threshold decision on the overall quality score, the candidate segments used to generate the dynamic cover are determined, thereby improving the click-through conversion rate and user dwell time of the generated cover, and supporting flexible adjustment of weights according to business needs to adapt to different scenarios.

[0102] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of the dynamic cover generation device 200 is given below. Further, please refer to... Figure 4 , Figure 4 This is a functional block diagram of a dynamic cover generation device 200 provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the dynamic cover generation device 200 provided in this embodiment are the same as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments. The dynamic cover generation device 200 includes: The acquisition module 201 is used to acquire multimodal data from the video stream.

[0103] The processing module 202 is used to obtain the cover value segment time series based on multimodal data; perform peak detection on the cover value segment time series to determine at least one highlight peak time point; obtain highlight event segments based on each highlight peak time point; and generate a dynamic cover based on the highlight event segments.

[0104] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 110 shown is either stored in or embedded in the operating system (OS) of the electronic device 100, and can be used by... Figure 1 The processor 120 executes the program. Meanwhile, the data and program code required to execute the above modules can be stored in the memory 110.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0106] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0107] If the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device 100, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating a dynamic cover, characterized in that, The method includes: Acquire multimodal data from the video stream; The cover value score time series was obtained based on the multimodal data; Peak detection is performed on the time series of the cover value to determine at least one highlight peak time point; Based on each of the aforementioned peak time points, a highlight event segment is obtained; A dynamic cover is generated based on the aforementioned highlight event fragment.

2. The dynamic cover generation method as described in claim 1, characterized in that, The steps for obtaining the cover value time series based on the multimodal data include: The multimodal data is input into a pre-trained cover value scoring model for processing to obtain the cover value score time series.

3. The dynamic cover generation method as described in claim 2, characterized in that, The training process of the cover value scoring model includes: Construct multiple comparison sample pairs, wherein each comparison sample pair includes a first sample and a second sample, wherein the first sample represents a video clip that is more suitable as a cover than the second sample; Each of the comparison samples is used to train the input preset model, and the training ends when the difference between the score of the first sample and the score of the second sample output by the preset model is greater than a preset boundary value. The resulting trained preset model is used as the cover value scoring model.

4. The dynamic cover generation method as described in claim 3, characterized in that, The step of constructing multiple comparison sample pairs includes: For any two candidate video segments in the video stream, identify the content implementation difficulty in each candidate video segment, and use the candidate video segment with higher content implementation difficulty as the first sample, and the candidate video segment with lower content implementation difficulty as the second sample; or, The bullet comments in each candidate video segment are counted. The candidate video segments that elicit a higher number of positive emotional bullet comments are selected as the first sample, and the candidate video segments that elicit a lower number of positive emotional bullet comments are selected as the second sample; or, For each candidate video segment, the human pose is estimated, and the candidate video segments with larger motion amplitudes are used as the first sample, and the candidate video segments with smaller motion amplitudes are used as the second sample; or, The system receives a user's selection operation for two candidate video segments, uses the selected candidate video segment as the first sample, and uses the candidate video segment that was not selected by the user as the second sample.

5. The dynamic cover generation method as described in claim 2, characterized in that, The multimodal data includes the original video frame sequence, the original audio signal, and the real-time bullet screen stream; the cover value scoring model includes a three-dimensional convolutional neural network, an audio encoder, a natural language processing module, and an evaluation module. The step of inputting the multimodal data into a pre-trained cover value scoring model to obtain a time series of cover value scores includes: The three-dimensional convolutional neural network is used to extract the pure visual features of the original video frame sequence; The acoustic features of the original audio signal are extracted using the audio encoder. The semantic features of the real-time bullet screen stream are extracted using the natural language processing module. The acoustic features and the semantic features are fused to obtain multimodal contextual features; The pure visual features and the multimodal contextual features are input into the evaluation module to obtain the cover value score time series.

6. The dynamic cover generation method as described in claim 1, characterized in that, The step of performing peak detection on the time series of the cover value to determine at least one highlight peak time point includes: Each local maximum point in the time series of the cover value is identified as a candidate peak. The candidate peaks are filtered and corrected by combining the semantic information obtained from the video stream, and the time point corresponding to each candidate peak that passes the correction is determined as the highlight peak time point.

7. The dynamic cover generation method as described in claim 1, characterized in that, The step of obtaining the highlight event fragment based on each highlight peak time point includes: Taking each of the aforementioned peak time points as the center, the process extends forward and backward along the time axis of the video stream until a preset stopping condition is met, thereby obtaining the start time point and end time point of the highlight event. The highlight event segment is obtained from the video stream based on the event start time and the event end time.

8. The dynamic cover generation method according to any one of claims 1 to 7, characterized in that, The step of generating a dynamic cover based on the highlight event fragment includes: The highlight event segments are subjected to duration normalization to obtain candidate segments; Obtain the peak score, average score, and duration features of the candidate segments; The overall quality score of the candidate segment is obtained based on the peak score, the average score, and the duration feature. If the overall quality score is not less than a preset threshold, a dynamic cover is generated based on the candidate segments.

9. A dynamic cover generation device, characterized in that, The device includes: The acquisition module is used to acquire multimodal data from the video stream; The processing module is used to obtain a cover value time series based on the multimodal data; perform peak detection on the cover value time series to determine at least one highlight peak time point; obtain highlight event segments based on each highlight peak time point; and generate a dynamic cover based on the highlight event segments.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the dynamic cover generation method according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dynamic cover generation method as described in any one of claims 1-8.

Images