Video highlight clip identification and training method of video highlight clip identification model

Abstract

The embodiment of the specification provides a video highlight segment identification method and a training method of a video highlight segment identification model. The scheme comprises the following steps: obtaining a target video to be identified; determining a plurality of video segments contained in the target video; determining a context representation of each video segment; inputting the context representation of each video segment into an attention branch of a pre-trained model to obtain an attention weight of each video segment; the attention weight is used to represent the aggregation weight of the corresponding video segment when generating the overall representation of the target video; inputting the context representation of each video segment into a reward branch of the pre-trained model to obtain a segment reward value of each video segment; the segment reward value is used to represent the business contribution degree of the corresponding video segment to the target video; fusing the attention weight and the segment reward value of each video segment to obtain a highlight score of each video segment; and determining a highlight segment from each video segment according to the highlight score of each video segment.

Description

Technical Field

[0001] This specification relates to the field of computer vision technology, and particularly to a method for video highlight segment recognition, a training method for a video highlight segment recognition model, and a verification method for a video highlight segment recognition model. This specification also relates to a video highlight segment recognition apparatus, a training apparatus for a video highlight segment recognition model, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In e-commerce advertising creative analysis, video editing, and content recommendation applications, it is often necessary to identify highly attractive or business-valuable segments (i.e., highlight segments) from complete advertising videos for secondary editing of advertising creatives, rapid generation of campaign materials, or automatic summarization of video content. Current technologies typically employ methods such as segmenting the video into segments, extracting the visual or audio features of each segment, and scoring and ranking each segment to determine the highlight segments. This enables compression of the complete video and rapid location of key content.

[0003] However, the existing technology still has room for improvement under the above processing methods. Therefore, there is a need to provide a more reliable solution for identifying video highlight segments. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a video highlight segment recognition method, a video highlight segment recognition model training method, a video highlight segment recognition model verification method, a video highlight segment recognition device, a video highlight segment recognition model training device, an apparatus, a computer-readable medium, and a computer program product, to provide a more reliable video highlight segment recognition solution.

[0005] According to a first aspect of one or more embodiments of this specification, a video highlight segment recognition method is provided, comprising: Obtain the target video to be identified; Identify the multiple video segments contained in the target video; Determine the contextual representation of each video segment; The context representation of each video segment is input into the attention branch of the pre-trained model to obtain the attention weight of each video segment; the attention weight is used to characterize the aggregate weight of the corresponding video segment when generating the overall representation of the target video; The context representation of each video segment is input into the reward branch of the pre-trained model to obtain the segment reward value of each video segment; the segment reward value is used to characterize the degree of business contribution of the corresponding video segment to the target video; The attention weight and the segment reward value of each video segment are fused to obtain the highlight score of each video segment; the highlight score increases with the increase of the attention weight and the increase of the segment reward value. Highlight segments are determined from each video segment based on the highlight score of each video segment.

[0006] According to a second aspect of one or more embodiments of this specification, a method for training a video highlight segment recognition model is provided, comprising: Obtain training sample videos and corresponding video-level business metrics; Identify the multiple video segments contained in the training sample video; Determine the contextual representation of each video segment; The context representation of each video segment is input into the attention branch of the model to be trained to obtain the attention weight of each video segment; the attention weight is used to characterize the aggregate weight of the corresponding video segment when generating the overall representation of the training sample video; The context representation of each video segment is input into the reward branch of the model to be trained to obtain the segment reward value of each video segment; the segment reward value is used to characterize the degree of contribution of each video segment to the video-level business metric. The segment reward values ​​of each video segment are weighted and aggregated according to the attention weight of each video segment to obtain the aggregation result; The network parameters of the model to be trained are updated based on the difference between the aggregation result and the video-level service metric.

[0007] According to a third aspect of one or more embodiments of this specification, a verification method for a video highlight segment recognition model is provided, comprising: Obtain the baseline video-level prediction score corresponding to the complete segment sequence; The input features of each video segment are set to zero in sequence to obtain the occlusion prediction score of each video segment after occlusion. The difference between the baseline video-level prediction score and each occlusion prediction score is used as the leave-one contribution for the corresponding video segment. The highlight segment recognition results are verified based on the ranking correlation between the leave-one contribution of each video segment and the highlight score of each video segment, and / or based on the overlap rate between the top K segments with the leave-one contribution and the top K segments with the highlight score, where K is a positive integer.

[0008] According to a fourth aspect of one or more embodiments of this specification, a video highlight segment recognition device is provided, comprising: The target video acquisition module is configured to acquire the target video to be identified. The video segment determination module is configured to determine multiple video segments contained in the target video; The context representation determination module is configured to determine the context representation of each video segment; The attention weight determination module is configured to input the context representation of each video segment into the attention branch of the pre-trained model to obtain the attention weight of each video segment; the attention weight is used to characterize the aggregate weight of the corresponding video segment when generating the overall representation of the target video; The segment reward value determination module is configured to input the context representation of each video segment into the reward branch of the pre-trained model to obtain the segment reward value of each video segment; the segment reward value is used to characterize the degree of business contribution of the corresponding video segment to the target video; The highlight score calculation module is configured to fuse the attention weight and the segment reward value of each video segment to obtain the highlight score of each video segment; the highlight score increases with the increase of the attention weight and the increase of the segment reward value. The highlight segment determination module is configured to determine highlight segments from each video segment based on the highlight score of each video segment.

[0009] According to a fifth aspect of one or more embodiments of this specification, a training apparatus for a video highlight segment recognition model is provided, comprising: The training sample and metric acquisition module is configured to acquire training sample videos and video-level business metrics corresponding to the training sample videos. The video segment segmentation module is configured to determine multiple video segments contained in the training sample video; The context representation module is configured to determine the context representation of each video segment; The attention weight prediction module is configured to input the context representation of each video segment into the attention branch of the model to be trained, and obtain the attention weight of each video segment; the attention weight is used to characterize the aggregate weight of the corresponding video segment when generating the overall representation of the training sample video; The segment reward value prediction module is configured to input the context representation of each video segment into the reward branch of the model to be trained to obtain the segment reward value of each video segment; the segment reward value is used to characterize the degree of contribution of each video segment to the video-level business metric. The aggregation module is configured to perform weighted aggregation of the segment reward values ​​of each video segment based on the attention weight of each video segment, and obtain the aggregation result; The parameter update module is configured to update the network parameters of the model to be trained based on the difference between the aggregation result and the video-level service metric.

[0010] According to a sixth aspect of one or more embodiments of this specification, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the computer instructions to implement the steps of the video highlight segment recognition method, the training method of the video highlight segment recognition model, or the verification method of the video highlight segment recognition model.

[0011] According to a seventh aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the video highlight segment recognition method, the training method of the video highlight segment recognition model, or the verification method of the video highlight segment recognition model.

[0012] According to an eighth aspect of one or more embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described video highlight segment recognition method, the training method of the video highlight segment recognition model, or the verification method of the video highlight segment recognition model.

[0013] One embodiment of this specification can achieve at least the following beneficial effects: The pre-trained model outputs attention weights and segment reward values ​​through two independent branches respectively—the former is only used to characterize the weight ratio of the segment in the overall video representation aggregation, and the latter is specifically used to characterize the contribution of the segment to the business objective. The physical meanings of the two are distinct from each other and do not interfere with each other. Then, the information from the two orthogonal dimensions is combined by fusion processing to obtain the highlight score. Since the dual-branch decoupling avoids the gradient conflict and physical meaning confusion caused by a single score simultaneously undertaking aggregation and discrimination functions, the highlight score is determined based on the representativeness of the segment in the overall video and its positive contribution to the business objective. Therefore, reliable identification of highlight segments in advertising videos can be achieved by using video-level business feedback signals without relying on manually labeled highlight tags. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This diagram illustrates an application scenario of a video highlight segment recognition method provided in the embodiments of this specification. Figure 2This is a schematic diagram of the structure of a video highlight segment recognition model provided in the embodiments of this specification; Figure 3 A flowchart illustrating a training method for a video highlight segment recognition model provided in an embodiment of this specification; Figure 4 A flowchart illustrating a verification method for a video highlight segment recognition model provided in an embodiment of this specification; Figure 5 A flowchart illustrating a video highlight segment recognition method provided in the embodiments of this specification; Figure 6 The embodiments provided in this specification correspond to Figure 3 A schematic diagram of the structure of a training device for a video highlight segment recognition model; Figure 7 The embodiments provided in this specification correspond to Figure 5 A schematic diagram of a video highlight segment recognition device; Figure 8 A structural block diagram of a computing device provided according to an embodiment of this specification is shown. Detailed Implementation

[0016] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0017] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0018] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “an,” “an,” “the,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification includes any or all possible combinations of one or more associated listed items.

[0019] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded.

[0020] Although the terms "first," "second," etc., may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first," without departing from the scope of one or more embodiments of this specification. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; often they are used to facilitate the distinction of objects. For example, "first server" and "second server" usually refer to two servers. To distinguish these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0021] Depending on the context, the word "if" as used here can be interpreted as "when," "when," or "in response to determination."

[0022] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0023] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0024] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0025] In e-commerce advertising creative analysis, video editing, and content recommendation applications, it is often necessary to identify highly attractive or business-valuable segments (i.e., highlight segments) from complete advertising videos for secondary editing of advertising creatives, rapid generation of campaign materials, or automatic summarization of video content. E-commerce advertising videos typically consist of multiple semantic segments, including an opening attraction, a description of pain points, product display, usage demonstration, selling point explanation, price promotion, trust verification, and action guidance.

[0026] Current video highlight detection technologies are mainly developing along two lines. One line is based on text matching, the core idea of ​​which is to input a text description and find the segment in the video that best matches that text. However, in practical applications, it is often impossible to pre-define a universal text that accurately describes the highlight content of a given advertising video—the highlight performance varies from video to video, and the perceived "highlights" of the same video differ from viewer to viewer. While the text matching-based research approach is common in academia, in real-world e-commerce advertising video processing scenarios, the diverse video content and varying highlight performances make it difficult to pre-define an accurate descriptive text for each video, thus limiting its applicability in industrial applications. The other line relies on manual annotation, where annotators manually select the time range of highlight segments after watching the video. However, the definition of highlight segments is highly subjective, and different annotators often disagree on the highlight judgments of the same video. Furthermore, segment-level manual annotation is costly and inefficient, making it difficult to meet the processing needs of large-scale advertising video data. In addition, some solutions attempt to determine highlight segments based on time alignment using multimodal information such as video frames, audio, and text. These solutions focus on visual saliency detection in general scenarios, but in e-commerce advertising scenarios where there is only click feedback data at the whole video level and a lack of segment-level annotation, it is difficult to effectively learn the actual contribution of each segment to the ad click effect.

[0027] In e-commerce advertising scenarios, video-level feedback data such as impressions, clicks, and consumption serve as available supervisory signals. A complete video can be viewed as a collection of multiple video segment instances (i.e., a multi-instance package), but it only has an overall package-level label (such as click-through rate), not instance-level labels for each segment. How to enable the model to recognize the contribution of each segment to the overall video effect, given only video-level feedback signals and no segment-level manual annotation, is a pressing technical problem that needs to be solved.

[0028] Therefore, a technical solution is needed that can identify highlight segments in an advertisement video by utilizing the feedback signal of the entire video in the absence of segment-level artificial highlight tags.

[0029] To address the shortcomings in related technologies, this specification provides a method for identifying video highlight segments in its embodiments. This method utilizes two independent branches of a pre-trained model to output attention weights and segment reward values. The attention weights output by the attention branch characterize the aggregation weight of the corresponding video segment in generating the overall representation of the target video (i.e., the numerical proportion of the segment in feature aggregation), while the segment reward values ​​output by the reward branch characterize the business contribution of the corresponding video segment to the target video. These two values ​​are then fused to obtain the highlight score for each video segment, thereby determining the highlight segment.

[0030] In the above scheme, since the attention weight and segment reward value are output by two independent branches of the same pre-trained model, their physical meanings are distinguished. The attention weight is only used to represent the numerical proportion of the segment during feature aggregation and does not bear the function of judging positive or negative contribution, while the segment reward value is specifically used to represent the degree of contribution of the segment to the business objective. This dual-branch decoupled architecture allows the two branches to learn independently and without interference—the attention branch only needs to focus on the representativeness of the segment features, and the reward branch only needs to focus on the relevance of the segment to the business objective—thus avoiding gradient conflicts and confusion of physical meaning caused by a single score simultaneously bearing the dual functions of aggregation weight and contribution criterion. At the same time, since the fusion processing integrates the two orthogonal dimensions of the aggregation weight of the segment in the overall video representation and the degree of contribution of the segment to the business objective, the final determined highlight segment is both representative in content and has a positive contribution to the business. Through the above dual-branch decoupled architecture and fusion processing mechanism, the segment reward value can maintain sufficient numerical discrimination in the unbounded real space, avoiding the convergence of rewards among segments due to the small magnitude of the supervision signal, thus outputting interpretable highlight scores even in the absence of segment-level manual labels. Therefore, this invention can automatically and reliably identify highlight segments in advertising videos using video-level business feedback signals without the need for manual highlight labeling.

[0031] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0032] Figure 1 This diagram illustrates an application scenario of a video highlight segment recognition method provided in the embodiments of this specification.

[0033] like Figure 1 As shown in the embodiments of this specification, the video highlight segment recognition method and system can be applied to the automated analysis and processing of e-commerce advertising videos. This technical solution utilizes a dual-branch architecture of a pre-trained model—the aggregation weight of the attention branch output segment in the overall video representation and the contribution of the reward branch output segment to the business objective—and then fuses the two to obtain a highlight score. This allows for the automatic location of highlight segments from advertising videos without manual annotation. This solution can be deployed on various computing devices, such as dedicated servers, cloud server clusters, computing nodes with GPU acceleration capabilities, edge computing devices, or terminal devices (such as personal computers, workstations, and smartphones).

[0034] In terms of deployment methods, flexible choices can be made based on the actual application scenario. For example, a pure cloud deployment method can be used, where the video is uploaded to a cloud server for highlight recognition processing and the results are returned; an edge-cloud collaborative method can also be used, where preprocessing such as video decoding, scene segmentation, and feature extraction is completed on the client side, while the cloud only performs model inference, thereby reducing data transmission volume; alternatively, the trained model can be directly deployed on a local computing device to complete highlight recognition without requiring an internet connection. Figure 1 This illustrates a scenario where a video is uploaded to a server 102 via a client 101 for highlight recognition, and the result is returned. However, this is not the only scenario in practical applications.

[0035] In an exemplary application scenario, content operators on an e-commerce platform need to quickly extract key clips from a 45-second skincare product advertisement video for use in their news feed. The operators upload the video to an ad creative analysis system. The system automatically segments the video into multiple clips, then uses a pre-trained model to calculate attention weights and reward values ​​for each clip, merging them to obtain a highlight score. After sorting the clips according to their highlight scores, the system displays three thumbnails of the video's highlights and their corresponding time intervals on the system interface, such as a product usage demonstration clip from 18 to 28 seconds and an effect comparison clip from 28 to 35 seconds. After previewing, the operators can directly export the selected highlight clips as independent video files for rapid generation of ad creatives. The entire process, from video upload to highlight clip output, can be completed in a short time, requiring no specialized technical knowledge from the operators, nor requiring manual viewing of the entire video or clip annotation.

[0036] This specification provides a training method for a video highlight segment recognition model, a video highlight segment recognition method, and a verification method for a video highlight segment recognition model. It also relates to a training device for a video highlight segment recognition model, a video highlight segment recognition device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0037] Figure 3 This is a flowchart illustrating a training method for a video highlight segment recognition model provided in an embodiment of this specification.

[0038] From a programming perspective, the entity executing the process can be a program hosted on an application server or application terminal. It can be understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities.

[0039] like Figure 3 As shown, the process may include the following steps: Step 202: Obtain training sample videos and corresponding video-level business metrics.

[0040] The training sample videos can be obtained from online advertising platforms, and the corresponding exposure, click-through rate, and cost feedback data for each video can be retrieved. When acquiring training data, an exposure threshold can be set to filter the data, retaining only video samples with exposure reaching a preset level. This ensures that the feedback data is statistically significant and avoids interference from random fluctuations caused by data sparsity due to low-exposure samples on model training.

[0041] In some embodiments, the same video content may correspond to multiple different video links due to repeated uploading, transcoding, or distribution. If data processing is directly based on video link identifiers, multiple different labels for the same video content will introduce training noise and may lead to data leakage due to the same content being distributed across training and validation sets. Therefore, a content identifier (canonical_video_id) can be generated using methods such as file hashing, keyframe-aware hashing (e.g., pHash), video embedding similarity, or business mapping relationships, mapping multiple video links corresponding to the same video content to the same content identifier. Then, basic business data such as exposure, clicks, and consumption are aggregated according to this content identifier to obtain the video-level business metrics corresponding to the training sample video. This process can reduce training noise caused by multiple conflicting labels for the same segment sequence.

[0042] In some embodiments, the video identifier in the main video address can be directly used as the content identifier to read a table file containing impressions, clicks, and consumption, and the sliced ​​video data files can be grouped according to the video identifier. If any valid video segment lacks a preset dimension of visual or audio embedding, the video sample can be removed to avoid confusion between zero vectors and real low-information segments.

[0043] The video-level business metrics refer to quantitative feedback data that reflects the overall performance of the advertising video, calculated from the impressions, clicks, and costs recorded by the advertising system. In some embodiments, these video-level business metrics may include Click-Through Rate (CTR), smoothed click-through rate, Cost Per Mille (CPM), Cost Per Click (CPC), and statistical transformation values ​​of the above metrics.

[0044] Let E be the total exposure, C be the total clicks, and S be the total cost. Then the original content click-through rate (CTR) is C / E, the cost per thousand exposures is 1000S / E, and the cost per click is S / max(C,1). To reduce the random fluctuations of low-exposure samples, a smoothed CTR can be used as a video-level business metric. The smoothed CTR is calculated as follows: y = (C + α×G) / (E + α), where y is the smoothed CTR, G is the global CTR of all content samples, and α is the smoothing coefficient. In one example, α can be 1000.

[0045] Based on some embodiments of this specification, the above-described content-level aggregation processing enables the feedback data of the same video content to be merged into a single training sample even if there are multiple delivery links, thus avoiding the introduction of training noise due to the same content corresponding to multiple tags. At the same time, through exposure threshold filtering and global click-through rate smoothing, the statistical fluctuations of low-exposure samples are effectively suppressed, making the quality of training data more reliable and providing a data foundation for the stable training of subsequent models.

[0046] In some embodiments, sample weights can be assigned to each training sample video to adjust the contribution of different samples to the model parameter updates. The sample weights can be calculated as follows: w = log(1 + E) / (1 + CV), where w is the sample weight, E is the exposure of the training sample video, and CV is the coefficient of variation of the click-through rate (CTR) of multiple video links for the same video content. The CTR is calculated as the ratio of the standard deviation to the mean of the CTRs of multiple video links corresponding to the same video content.

[0047] In some embodiments, when there is no data with multiple links to the same content, the sample weight can be directly taken as log(1+E).

[0048] Based on some embodiments in this specification, assigning greater weight to samples with higher exposure through the log(1+E) term can enhance the dominant role of high-confidence samples in training. Simultaneously, reducing the weight of samples with large fluctuations in cross-link click-through rates through the 1 / (1+CV) term can suppress training interference caused by differences in distribution channels or data noise. This weighting strategy comprehensively measures sample quality from two dimensions: data volume and data stability. This allows the model to focus more on high-confidence samples during training, thereby improving the model's robustness and generalization ability.

[0049] Step 204: Determine the multiple video segments contained in the training sample video.

[0050] In some embodiments, after acquiring training sample videos, scene detection is performed on the training sample videos to determine initial video segments divided by scene. Scene detection can be implemented based on shot transition detection—when there is a significant change in the visual content between adjacent frames (such as a sudden change in color histogram, an optical flow change exceeding a threshold, etc.), the location is marked as a scene boundary, thereby dividing the video into multiple initial video segments.

[0051] The number and duration of segments in different videos often vary. Directly using the original scene segmentation results for subsequent processing may lead to excessive differences in segment duration, introducing sorting bias and hindering batch tensor computation. Therefore, in some embodiments, the initial video segments are adaptively processed based on their durations to obtain multiple video segments for subsequent model input. The adaptive processing includes merging, retention, or secondary segmentation. Specifically: for initial video segments with durations less than the minimum duration threshold, they are merged with adjacent scenes, and the duration of the merged scene does not exceed the maximum duration threshold; initial video segments with durations between the minimum and maximum duration thresholds are retained; for initial video segments with durations greater than the maximum duration threshold, overlapping secondary segmentation is performed within the segment according to a preset window length and a preset step size to obtain multiple sub-segments.

[0052] In one example, the minimum duration threshold can be set to 1 second, the maximum duration threshold can be set to 6 seconds, the preset window length for long scenes can be set to 4 seconds, and the preset step size can be set to 2 seconds. For initial video segments exceeding 6 seconds in length, an overlapping sliding window with a window length of 4 seconds and a step size of 2 seconds is used for segmentation, thereby splitting the long segment into multiple sub-segments with overlapping parts, which preserves the temporal continuity of the segments while controlling the duration range of individual segments.

[0053] To facilitate batch tensor computation, in some embodiments, a maximum number of segments can be preset (e.g., a maximum of 40 video segments). When the actual number of video segments obtained exceeds this maximum number, the excess is truncated; when the actual number of video segments obtained is less than this maximum number, zero-padding is used to pad to the maximum number of segments. Simultaneously, a valid segment mask is generated, assigning a first mask value (e.g., 1) to the real video segments and a second mask value (e.g., 0) to the padded segments. In subsequent processing, the padded segments do not participate in attention weight calculation, reward mean calculation, or gradient backpropagation of the loss function.

[0054] Based on some embodiments of this specification, through the above-described scene detection and adaptive segmentation processing, excessively short scenes are merged and excessively long scenes are broken down, so that the duration of each video segment is controlled within a reasonable range, avoiding deviations in highlight sorting due to excessive differences in scene duration; at the same time, by fixing the number of segments to a preset maximum value and generating effective segment masks, variable-length videos can be input into the model in a unified tensor form for batch processing, providing a standardized data foundation for subsequent model training and inference.

[0055] Step 206: Determine the contextual representation of each video segment.

[0056] In some embodiments, when determining the contextual representation of each video segment, input features for each video segment are first extracted. Input features may include at least one of visual features, audio features, segment character features, and temporal numerical features.

[0057] Visual features can reflect the image content information of a video clip. In some embodiments, image features of each frame in a video clip can be extracted, and then average pooling or max pooling can be performed on the features of each frame within the clip to obtain the visual embedding of the clip. In one example, the visual embedding can have a dimension of 2048. Audio features can reflect the sound information of a video clip, such as background music and human voices. In one example, the audio embedding can have a dimension of 1024.

[0058] Segment role features are used to characterize the semantic role that a corresponding video segment plays in the video narrative structure of the training sample videos. E-commerce advertising videos typically consist of multiple segments with different narrative functions, such as opening attraction, pain point description, product display, usage demonstration, selling point explanation, price promotion, trust verification, comparison display, action guidance, brand identification, and transitions. Different advertising videos may contain some or all of the above role types. In some embodiments, segment role features can be probability vectors, where each element corresponds to a preset role label (such as opening attraction, pain point description, product display, etc.). The values ​​of each element are independent and range from 0 to 1, representing the probability that the video segment belongs to the corresponding preset role label. The probabilities of each role are independent without performing Softmax normalization, which allows a segment to have multiple role attributes simultaneously—for example, a segment can be both a "usage demonstration" and a "selling point explanation," or it can contain both "brand identification" and "action guidance."

[0059] The probabilities of characters in a video clip can be generated in various ways. In some embodiments, the video clip's visuals, speech-recognized text, and optical character recognition (OCR) text can be input into a multimodal large model (such as a visual language model), which then outputs the independent probabilities of each preset character label. In other embodiments, character recognition and probability generation can also be performed using methods based on manual rules or dedicated classifiers.

[0060] Temporal numerical features are used to characterize at least one of the position and duration of a corresponding video segment on the timeline of the training sample video. In some embodiments, temporal numerical features can be represented as a four-dimensional vector [duration, position_ratio, is_long_scene_subsegment, start_time], where duration is the segment duration, position_ratio is the proportion of the segment's starting position to the total video duration (i.e., start_time divided by video_total_duration), is_long_scene_subsegment is an identifier indicating whether it is a sub-segment obtained from a secondary segmentation of a long scene, and start_time is the segment's start time. This temporal information allows the model to perceive the relative position and duration distribution of each segment on the overall video timeline.

[0061] In some embodiments, after extracting the input features of each video segment, the input features are fed into the encoding network of the model to be trained to obtain the contextual representation of each video segment. The encoding network may include a segment encoder and a cross-segment encoder.

[0062] The segment encoder, in this context, refers to the first-level encoding component of the encoding network in the pre-trained model. It is configured to perform intra-modal projection and multimodal fusion on the input features of a single video segment, outputting the segment's latent representation. The segment encoder processes each video segment independently, without passing information between segments. Specifically, the segment encoder can perform intra-modal projection on the input features of different modalities (e.g., projecting visual and audio embeddings to a unified latent space through linear layers, mapping temporal numerical features to a numerical feature space, and mapping character probability vectors to character embeddings), then concatenate the projected multimodal features and fuse them through a multilayer perceptron to obtain the segment's latent representation.

[0063] A cross-segment encoder is a second-level encoding component in the encoding network of a pre-trained model. It is configured to receive the segment latent representations of each video segment, model the temporal dependencies and semantic collaborations between segments through attention interactions, and output the contextual representations of each video segment. The cross-segment encoder ensures that the contextual representation of each segment not only includes the content information of the segment itself but also incorporates semantic association information with other segments in the same video. In some embodiments, the cross-segment encoder can adopt a Transformer encoder structure, utilizing a self-attention mechanism to achieve global cross-segment information interaction.

[0064] In some embodiments, a segment encoder and a cross-segment encoder are sequentially connected to form an encoding network. The multimodal input features of each video segment are first processed by the segment encoder to obtain the segment latent representation, and then by the cross-segment encoder to obtain the context representation. The segment encoder processes each segment independently and is unaware of the temporal relationships between segments; the cross-segment encoder, on the other hand, establishes global dependencies between segments through a self-attention mechanism, so that the output context representation contains not only the content information of the segment itself, but also incorporates semantic associations and temporal coordination information with other segments in the video.

[0065] In practical applications, firstly, the input features of each video segment are input into a segment encoder to obtain the segment latent representation of each video segment. In some embodiments, visual embeddings and audio embeddings are projected onto a preset latent space through linear layers; temporal numerical features are mapped to a lower-dimensional numerical feature space through linear layers; and the character probability vector is multiplied by a trainable character embedding matrix to obtain the character embedding. Then, the projected visual features, audio features, temporal numerical features, and character embeddings are concatenated, and the concatenated features are input into a segment multilayer perceptron to obtain the initial latent representation of each segment. This segment multilayer perceptron may sequentially include a first linear layer, a layer normalization layer, a GELU activation layer, a Dropout layer, a second linear layer, and a second layer normalization layer. In one example, the latent dimension can be set to 256 dimensions, and the Dropout ratio can be set to 0.1.

[0066] In some embodiments, the segment latent representations of each video segment are input into a cross-segment encoder to obtain the context representations of each video segment. The cross-segment encoder can employ a Transformer encoder structure, which can model the temporal dependencies and semantic cooperation relationships between segments. In some embodiments, the cross-segment encoder can use an effective segment mask as a key padding mask to avoid padding positions participating in self-attention computation and context aggregation. In one example, the cross-segment encoder can employ a 1-layer Transformer encoder, 4 attention heads, and a 1024-dimensional feedforward network.

[0067] Based on some embodiments of this specification, by extracting multi-dimensional input features such as visual, audio, role, and temporal information, the model can simultaneously acquire content information (images and sounds), semantic information (role function in the advertising narrative), and structural information (position and duration on the video timeline) of a segment, providing a rich information foundation for subsequent segment representation learning. In particular, the role features adopt the form of mutually independent probability vectors, enabling a segment to simultaneously assume multiple semantic roles, which is more in line with the actual situation of overlapping segment functions in e-commerce advertising videos. By fusing multimodal features into segment latent representations through a segment encoder, and then modeling the contextual relationships between segments through a cross-segment Transformer encoder, the representation of each segment not only contains its own content information, but also incorporates temporal associations and semantic coordination information with preceding and following segments, thereby improving the accuracy and contextual consistency of segment representation.

[0068] Figure 2 This is a schematic diagram of the structure of a video highlight segment recognition model provided in the embodiments of this specification.

[0069] See Figure 2 Overall, the model represents an advertisement video as a sequence of N segments. Each segment is first encoded by a multimodal projection and a segment encoder, and then the contextual relationships between segments are modeled through a cross-segment encoder. The segment encoder and the cross-segment encoder form an encoding network, with attention and reward branches connected to the output of this network. Subsequently, the model outputs attention weights and segment reward values ​​in parallel. Finally, the highlight score is determined based on the attention weights and segment reward values.

[0070] Step 208: Input the context representation of each video segment into the attention branch of the model to be trained to obtain the attention weight of each video segment; the attention weight is used to characterize the aggregate weight of the corresponding video segment when generating the overall representation of the training sample video.

[0071] In some embodiments, the attention branch maps the context representation of each video segment to a raw attention value (scalar), which characterizes the unnormalized saliency of the corresponding segment in the temporal context. This raw attention value is then normalized using a temperature scaling parameter to obtain the attention weight for each video segment.

[0072] The attention weights can be calculated using the Softmax function with a temperature parameter: a_i = exp(u_i / τ) / Σ_j exp(u_j / τ), where u_i is the original attention value for the i-th video segment, and τ is the temperature scaling parameter. u_i is non-linearly amplified using the exponential function exp and divided by the sum of the exponents of all valid segments, mapping u_i to the (0,1) interval, while ensuring that the sum of the weights for all segments equals 1.

[0073] The above summation is only performed on valid video segments. The original attention value of the filler segment can be set to the minimum value that the data type can represent before normalization, so that the attention weight obtained by the filler segment after normalization is close to zero, which does not affect the weight distribution of valid segments.

[0074] The temperature scaling parameter τ can be set as a learnable parameter and constrained during training to be no less than a preset positive number (e.g., 0.1). When τ is large, the attention weights of each segment tend to be evenly distributed, indicating that the model treats all segments equally; when τ is small, the attention weights tend to be one-hotly distributed (i.e., concentrated on a few segments), indicating that the model only focuses on the segments with the strongest response. It should be noted that the above temperature scaling parameter can also be equivalently defined as the inverse temperature form, that is, normalized using u_i × β (where β = 1 / τ).

[0075] In some embodiments, the attention branch may employ a single linear layer to map the context representation to scalar attention primitive values. The output of this linear layer may be initialized with a small gain to reduce numerical instability in the early stages of training.

[0076] Based on some embodiments of this specification, by employing a learnable temperature scaling parameter instead of a fixed temperature, the model can adaptively adjust the concentration of attention distribution according to different video content during training. For fast-paced promotional videos with concentrated climaxes, the model can make the attention distribution sharper to focus on key segments; for presentation videos with smooth content and multiple segments of similar importance, the model can make the attention distribution smoother to evenly focus on each segment. Simultaneously, by imposing a constraint on the temperature parameter to be no lower than a preset positive number, numerical instability and overly sharp attention distribution caused by temperatures approaching zero can be avoided. This mechanism allows the allocation of attention weights to adapt to the differences in the content rhythm of different videos, providing a more reasonable weight basis for subsequent multi-instance aggregation.

[0077] Step 210: Input the context representation of each video segment into the reward branch of the model to be trained to obtain the segment reward value of each video segment; the segment reward value is used to characterize the degree of contribution of each video segment to the video-level business metric.

[0078] In e-commerce advertising scenarios, the available supervisory signals are video-level business metrics (such as click-through rate) for the entire video, but it is impossible to obtain individual contribution tags for each segment. Therefore, the embodiments in this specification treat the advertising video as a multi-instance bag, with each video segment as an instance within the bag. Through multi-instance learning, the reward value of the supervised segment is learned using video-level tags.

[0079] In some embodiments, the segment reward values ​​of each video segment are weighted and aggregated according to the attention weight of each video segment to obtain the aggregated result m = Σ_i a_i × r_i, where a_i is the attention weight of the i-th segment and r_i is the segment reward value of the i-th segment. This aggregated result represents the model's prediction of the overall business performance of the video.

[0080] When using video-level business metrics (such as click-through rate y) as a supervisory signal, if the aggregated result m is directly aligned with the original click-through rate y (which ranges from 0 to 1, typically between 0.5% and 5% for e-commerce ads), then m, as a weighted average of r_i, will be constrained to the low-value range of y. Since r_i is in an unbounded real space while the supervisory target y is in a (0,1) probability space, to force the weighted aggregated result to approach a small value close to zero, the model needs to synchronously compress the r_i of all segments to a very small range close to zero. This will cause segments with positive and negative contributions to be unable to form an effective distinction in reward value, resulting in the collapse of the reward branch. At the same time, the highlight score is obtained by multiplying the attention weight by the segment reward value. When the reward value is generally close to zero, the highlight score also tends to zero, which will weaken the effective gradient backpropagation of the attention branch.

[0081] To avoid the aforementioned problems, in some embodiments, the step of inputting the context representation of each video segment into the reward branch of the model to be trained to obtain the segment reward value of each video segment includes: the reward branch maps the context representation of each video segment to a segment reward value in an unbounded real number space; this segment reward value is used to characterize the degree of contribution of the corresponding video segment to the video-level service metrics. Correspondingly, the step of updating the network parameters of the model to be trained based on the difference between the aggregation result and the video-level service metrics includes: calculating the aggregation loss based on the difference between the aggregation result and the log-odds transform value of the video-level service metrics; and updating the network parameters of the model to be trained based on the aggregation loss.

[0082] Specifically, video-level business metrics (such as smoothed click-through rate y) are mapped to an unbounded real number space using a logit transformation, resulting in the supervision target t = log(y / (1-y)). In actual calculations, to avoid numerical overflow, a numerically stable form with a small constant ε can be used: t = log(y+ε) - log(1-y+ε). This supervision target and the segment reward value reside in the same unbounded real number space. At this point, the aggregation result m = Σ_i a_i × r_i is aligned with the supervision target t in the same numerical space. r_i does not need to be compressed into the low range of click-through rate; different segments can freely differentiate within the unbounded real number space—segments with larger positive contributions can take larger positive values, while segments with smaller or negative contributions can take smaller or negative values, thus maintaining the numerical differentiation between segments.

[0083] Then, the aggregation loss is calculated based on the difference between the aggregation result and the supervision target. In some embodiments, Huber loss can be used as the aggregation loss function, i.e., L_mil = Huber(m, t). Huber loss combines the advantages of mean squared error and mean absolute error, has a certain robustness to outliers, and can provide more stable gradients during training. The network parameters of the model to be trained are updated according to this aggregation loss.

[0084] In some embodiments, the reward branch can be implemented using a two-layer perceptron, mapping the context representation to fragment reward values ​​without sigmoid compression. The weights of the reward output layer can be initialized with a small gain, and the bias can be initialized to zero to reduce numerical instability in the early stages of training.

[0085] Based on some embodiments of this specification, the reward branch directly outputs unbounded real numbers as segment reward values, while the video-level business metrics are mapped to the unbounded real number space through log-probability transformation as supervision targets. The two are aligned in the same numerical space, and the segment reward values ​​do not need to be compressed to the low value range of the click-through rate to complete the consistency learning with the video-level targets. This allows segments with positive and negative contributions to form sufficient differentiation in reward values, and the gradient signal corresponding to the aggregation loss can be effectively backpropagated to the reward branch, maintaining the numerical distinguishability of segment rewards and avoiding the problems of reward collapse and gradient vanishing.

[0086] Step 212: Based on the attention weight of each video segment, the segment reward value of each video segment is weighted and aggregated to obtain the aggregation result.

[0087] In some embodiments, after obtaining the attention weight a_i and segment reward value r_i of each video segment, the segment reward values ​​of each video segment are weighted and aggregated according to the attention weight of each video segment to obtain the aggregation result m = Σ_i a_i × r_i.

[0088] The aggregation result can be understood as the model's predicted score for the overall business performance of the training sample video, based on the attention weights and reward branch outputs of each current segment. By compressing the discrete information of multiple segments into a single video-level scalar value, it lays the foundation for subsequent alignment with video-level supervision targets.

[0089] Based on some embodiments of this specification, since the attention weights satisfy the condition that the sum of all valid segments is 1, the weighted aggregation result m has a clear mathematical meaning—it is the expected value of the reward value of each segment under the attention distribution, which reflects both the overall level of the segment reward value and the differentiated influence degree given to different segments through the attention weights; the aggregation result summarizes the segment-level information into a single scalar at the video level, so that the prediction score of the entire video can be directly compared with the video-level business indicators, thereby realizing the information transfer from the instance level to the package level, and providing a computable supervision interface for subsequent weakly supervised training.

[0090] Step 214: Update the network parameters of the model to be trained based on the difference between the aggregation result and the video-level service metric.

[0091] In some embodiments, after obtaining the aggregation result *m*, a multi-instance aggregation loss *L_mil* is calculated based on the difference between the aggregation result and the video-level business metric. The multi-instance aggregation loss can take the form of Huber loss: *L_mil = Huber(m, t)*, where *t* is the supervised target of the video-level business metric (such as smoothed click-through rate) after log-odds transformation. The network parameters of the model to be trained are updated based on this multi-instance aggregation loss—the gradient is simultaneously backpropagated to both the attention branch and the reward branch by this loss, enabling the parameters of the two branches to coordinate and optimize together during training. Although the attention branch and the reward branch share the encoding network, they output values ​​with different physical meanings, thereby achieving functional independence and decoupling.

[0092] In some embodiments, the video-level business metric includes smoothed click-through rate (CTR); the smoothed CTR is calculated based on the exposure, clicks, and global CTR of the training sample videos.

[0093] Let E be the total exposure and C be the total clicks for a training sample video. The original content click-through rate (C / E) is then C / E. To reduce the random fluctuations of low-exposure samples, a global C / E (the overall average C / E of all content samples) is introduced as prior knowledge. The smoothed C / E is calculated as follows: y = (C + α × G) / (E + α), where α is the smoothing coefficient. In one example, α can be 1000. When the exposure is low, the smoothed C / E will converge towards the global C / E, thus avoiding statistical noise caused by low exposure. When the exposure is high, the smoothed C / E will approach the original C / E, with the sample's own click performance dominating this metric.

[0094] Based on some embodiments of this specification, the original click-through rate (CTR) is smoothed using Bayesian methods by introducing a global CTR as a prior. When the sample exposure is low, the smoothed CTR approaches the global average level, avoiding interference from extreme values ​​caused by data sparsity. When the sample exposure is high, the smoothed CTR approximates the original CTR of the sample itself, preserving the true performance information of the sample. This mechanism ensures that the CTR, as a supervisory target, maintains statistical stability under different exposure levels, providing a more reliable supervisory signal for model training.

[0095] In some embodiments, multiple auxiliary tasks can be introduced to improve model training performance. In some embodiments, after obtaining the video representation (i.e., the result of weighted summation of attention weights and context representation), the video representation can be input into multiple auxiliary prediction heads to predict auxiliary business metrics for the training sample videos. After obtaining the video representation by weighted summation of attention weights and context representation, the video representation can be input into multiple prediction heads, such as a click-through rate (CPM) prediction head, a cost-per-thousand (CPM) prediction head, a relative global CPM improvement prediction head, and a cost-per-click (CPC) prediction head, to obtain predicted values ​​for each auxiliary business metric. Auxiliary business metrics can include at least one of cost-per-thousand (CPM), cost-per-click (CPC), and relative global CPM improvement (Lift). Specifically, for the main task of CPM prediction, the prediction head can use a Sigmoid activation function to map the output to a (0,1) probability space; for long-tailed economic metrics such as cost-per-thousand (CPM), cost-per-click, and relative global CPM improvement, regression can be performed in log space, i.e., the prediction target is log(1 + metric value), to balance the difference in numerical magnitude. Then, auxiliary losses are calculated between the predicted and actual values ​​of each auxiliary service indicator; for example, Huber loss can be used. These auxiliary losses, along with the multi-instance aggregation loss, constitute part of the total loss and are used together to update the network parameters of the model being trained. The backpropagation of auxiliary losses also involves the shared coding network, thus further improving the quality of the video representation.

[0096] In a specific example, for the click-through rate (CTR) prediction master task, the video representation can be input into the CTR prediction head. The head's output is mapped to the (0,1) probability space using a Sigmoid activation function, yielding the predicted CTR p_ctr. Then, the video-level CTR master task loss L_ctr = Huber(p_ctr, ctr_smooth), where ctr_smooth is the smoothed CTR, is calculated. This master task loss allows the model to directly learn video-level CTR predictions, complementing the multi-instance aggregation loss—the former directly regresses from the overall video representation, while the latter indirectly regresses from the aggregation of segment-level reward values.

[0097] In some embodiments, the training method of the video highlight segment recognition model further includes: predicting auxiliary business indicators of the training sample video based on the aggregation result; the auxiliary business indicators include at least one of the following: cost per thousand exposures, cost per click, and relative global click-through rate improvement; calculating the auxiliary loss between the predicted value and the true value of the auxiliary business indicators; and updating the network parameters of the model to be trained based on the auxiliary loss.

[0098] The cost per thousand impressions is calculated as 1000 × S / E, reflecting the cost of acquiring traffic for the video; the cost per click is calculated as S / max(C, 1), reflecting the cost efficiency of acquiring users for the video; and the improvement in relative global click-through rate is calculated as log(y + ε) - log(G + ε), reflecting the video's click-through rate advantage relative to the overall average level, where y is the smoothed click-through rate, G is the global click-through rate, and ε is a small constant used to avoid numerical overflow in logarithmic operations.

[0099] For long-tailed economic indicators such as cost per thousand impressions and cost per click, regression can be performed in log space, with the prediction target being log(1 + indicator value) to balance the numerical differences between samples of different magnitudes. Then, the auxiliary loss between the predicted and true values ​​of each auxiliary business indicator is calculated. This auxiliary loss, together with the multi-instance aggregation loss, is used to update the network parameters of the model to be trained.

[0100] During training, the above losses can be weighted and summed according to their respective weight coefficients to obtain the joint loss: L_total = λ_ctr×L_ctr +λ_mil × L_mil + λ_cpm × L_cpm + λ_lift ×L_lift + λ_cpc × L_cpc + λ_s × L_s + λ_e × L_e + λ_r × L_spread. Then, a backpropagation is performed based on this joint loss, while updating the network parameters of the entire model to be trained (including the shared encoder network, attention branch, and reward branch). In one example, the weights of each loss term can be set as: λ_ctr = 1.0, λ_mil = 0.1, λ_cpm = 0.3, λ_lift = 0.1, λ_cpc = 0.1, λ_s = 0.005, λ_e = 0.05, λ_r = 0.01.

[0101] During training, the AdamW optimizer can be used for parameter updates, cosine annealing can be used to adjust the learning rate, and gradient norm pruning and early stopping mechanisms can be used to improve training stability. In GPU-enabled training environments, automatic mixed precision can be used to accelerate training.

[0102] For splitting the training and validation sets, the data can be divided according to the video content identifiers, and a fixed splitting file can be saved. During evaluation, only this splitting file is read, without re-randomizing the data, to ensure the stability and reproducibility of the training and validation data.

[0103] Based on some embodiments of this specification, by simultaneously backpropagating the multi-instance aggregation loss to the attention branch and the reward branch, the two branches learn different physical meanings while sharing the same encoding network—the attention branch learns the weight ratio of the segment in feature aggregation, and the reward branch learns the contribution of the segment to the business objective—thus achieving functional independence and decoupling. By introducing auxiliary prediction tasks such as cost per thousand exposures, cost per click, and relative global click-through rate improvement, the video representation needs to support multiple prediction objectives simultaneously. The shared encoding network is forced to learn more generalized feature representations, thereby improving the model's comprehensive prediction capability on different business metrics and providing a more robust segment representation for highlight ranking in the subsequent inference stage.

[0104] In some embodiments, to prevent the segment reward values ​​of each video segment from becoming too similar and losing their discriminative power, a reward dispersion loss can be introduced. Specifically: the variance Var(r) of the segment reward value of each video segment is calculated; when the variance is lower than a preset minimum variance threshold v_min, the reward dispersion loss is calculated based on the difference between the variance and the preset minimum variance threshold; when the variance reaches or exceeds the preset minimum variance threshold, the reward dispersion loss is zero; and the network parameters of the model to be trained are updated based on the reward dispersion loss.

[0105] For example, the reward spread loss can take the form of a hinged loss: L_spread = max(0, v_min -Var(r)). Here, the preset minimum variance threshold v_min can be set to 1.0. This loss only penalizes insufficient reward variance, stopping further variance growth once the threshold is reached, thus encouraging fragment reward differentiation while avoiding training instability caused by unbounded variance growth.

[0106] Based on some embodiments of this specification, the distribution of segment reward values ​​is constrained by a variance lower bound hinge loss. When the difference in reward values ​​between segments is insufficient, a penalty is applied to widen the gap, enabling effective differentiation between different segments in terms of reward values. When the difference is sufficient, the loss is zero, avoiding training instability caused by continuously pushing the variance to expand unbounded. This constraint provides a guarantee for the discriminativeness of segment rewards from the perspective of numerical distribution, ensuring that segments with positive and negative contributions have clear numerical expressions in terms of reward values.

[0107] In some embodiments, to prevent attention weights from becoming too evenly distributed and losing focus, an attention entropy constraint loss can be introduced. Specifically: the entropy of the attention weights is normalized based on the number of valid video segments among the plurality of video segments; when the normalized entropy is lower than a preset lower limit, the attention entropy constraint loss is calculated based on the difference between the normalized entropy and the preset lower limit; and the network parameters of the attention branch are updated based on the attention entropy constraint loss.

[0108] The entropy H of the attention weights can be calculated as H = -Σ_i a_i × log(a_i), where a_i is the attention weight of the i-th valid video segment. Dividing the entropy H by the logarithm of the number of valid video segments, log(N_eff) (where N_eff is the number of valid video segments), yields the normalized entropy. When the normalized entropy is below a preset lower limit, the attention entropy constraint loss is calculated based on the difference between the normalized entropy and the preset lower limit; when the normalized entropy reaches or exceeds the preset lower limit, the attention entropy constraint loss is zero. This constraint ensures that the attention distribution maintains a certain degree of concentration—neither too evenly distributed (equal weights for all segments leading to a lack of focus) nor too concentrated (a single segment with a weight of 1 leading to overfitting).

[0109] Based on some embodiments of this specification, the attention weight distribution is kept at a reasonable level of concentration by normalizing the attention entropy constraint—a penalty is imposed when the attention is too even to encourage the model to focus on key segments and avoid losing the ability to locate highlights due to the inability to distinguish the importance of segments—while normalization is performed by the number of effective segments so that the constraint has a consistent constraint scale for videos of different lengths.

[0110] In some embodiments, to take advantage of the temporal continuity of the video, an adjacent attention smoothing loss can be introduced. Specifically: the difference (e.g., absolute difference) between the attention weights of temporally adjacent valid video segments among the plurality of video segments is calculated; the adjacent attention smoothing loss is calculated based on the difference; and the network parameters of the attention branch are updated based on the adjacent attention smoothing loss.

[0111] The adjacent attention smoothing loss can be implemented using L1 regularization: L_s = (1 / (N_eff - 1)) × Σ_{i=1}^{N_eff-1} |a_{i+1} - a_i|, where N_eff is the number of valid video segments, and a_i is the attention weight of the i-th valid video segment. This loss penalizes large differences in the attention weights of adjacent segments, prompting a smooth change in the attention weights of adjacent segments and avoiding drastic jumps, thus conforming to the natural gradual change of video content in time.

[0112] Based on some embodiments of this specification, by constraining the change range of attention weights of adjacent segments through adjacent attention smoothing loss, the attention distribution is made to maintain a smooth transition along the time axis, avoiding unreasonable highlight localization caused by drastic jumps in attention weights of adjacent segments. At the same time, it suppresses the overfitting of the model to training data noise to a certain extent and improves the temporal rationality of highlight ranking in the inference stage.

[0113] While one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible execution order among many steps and does not represent the only possible execution order. The order of some steps may be adjusted according to actual needs, or some steps may be omitted. When the claims involve method steps, changes in the order of such steps, or parallel execution between steps, are also within the scope of protection of the claims.

[0114] Figure 3 The method described above acquires training sample videos and corresponding video-level business metrics. After determining the context representation of each video segment, the context representation is input into the attention branch and reward branch of the model to be trained. The attention branch outputs attention weights, which characterize the aggregation weights of segments when generating the overall representation, and the reward branch outputs segment reward values, which characterize the contribution of segments to the video-level business metrics. The reward values ​​of each segment are then aggregated according to the attention weights to obtain the aggregation result. Finally, the network parameters are updated based on the difference between the aggregation result and the video-level business metrics. Since the attention branch and reward branch learn different physical meanings on the basis of sharing the same encoding network (the attention branch learns the weight ratio of segments in feature aggregation, and the reward branch learns the contribution of segments to the business objectives), they are jointly updated and coordinated under the supervision of the same loss function. This allows the model to learn the potential contribution of each segment to the overall business performance of the video using only video-level feedback signals without segment-level manual annotation, providing a trainable basic model for the subsequent recognition of highlight segments.

[0115] Building upon the aforementioned training framework, the model training process was further refined and enhanced from multiple dimensions: In feature construction, multimodal input features such as visual, audio, segment roles, and temporal values ​​were extracted and processed through an encoding network to obtain contextual representations. This ensures that each segment's representation simultaneously includes multi-dimensional information such as visual content, audio information, narrative semantic roles, and position and duration on the timeline, providing a richer input foundation for subsequent dual-branch computations. Through a two-level encoding structure of segment encoders and cross-segment encoders, each segment, in addition to obtaining its own content representation, further integrates temporal associations and semantic collaboration information with preceding and following segments. The attention branch normalizes the original attention value using a learnable temperature scaling parameter, enabling the model to adapt to different video content segments. The algorithm adaptively adjusts the concentration of attention distribution; the reward branch maps the context representation to fragment reward values ​​in an unbounded real space and aligns the aggregation result with the log-odds transformation value of video-level business metrics, so that the fragment reward value does not need to be compressed into the low value range of the click-through rate to complete the consistency learning with the video-level target, maintaining the numerical distinguishability of the reward value; in terms of supervised signal processing of training data, the original click-through rate is smoothed by introducing the global click-through rate, reducing the impact of statistical fluctuations of low-exposure samples on the quality of the supervised signal; by introducing auxiliary prediction tasks such as cost per thousand exposures, cost per click, and relative global click-through rate improvement, the video representation needs to support multiple business objectives simultaneously, and the shared coding network can learn more generalized feature representations under the joint supervision of multiple tasks.

[0116] In terms of constraint mechanisms, the reward dispersion loss constrains the variance of segment reward values ​​through a variance lower bound hinge. It only applies penalties to widen the gap when the reward difference is insufficient, and stops penalizing after reaching a threshold to avoid unbounded variance growth, thus ensuring sufficient differentiation of reward values ​​between segments. The attention entropy constraint normalizes the entropy of attention weights to ensure that the attention distribution maintains a reasonable degree of concentration—avoiding the model's inability to focus on key segments due to attention averaging, and also avoiding overfitting due to excessive concentration. The adjacent attention smoothing loss constrains the magnitude of attention weight changes between temporally adjacent segments, ensuring that the attention distribution maintains a smooth transition along the time axis, conforming to the natural gradual change of video content in time sequence, and suppressing overfitting to training data noise. During training, the multi-instance aggregation loss, reward dispersion loss, attention entropy constraint loss, and adjacent attention smoothing loss are weighted and summed according to their respective weight coefficients to obtain the joint loss. This joint loss updates the network parameters of the entire model to be trained in one backpropagation, so that the various constraints work together in training. This ensures the model's accuracy in fitting video-level business indicators, as well as the numerical discrimination of segment reward values, the reasonable concentration of attention distribution, and the temporal continuity, thereby improving the stability of model training and the quality of output highlight segments.

[0117] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they have not been described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0118] Based on the same idea, the embodiments of this specification also provide a verification method for the video highlight segment recognition model corresponding to the training method of the above-mentioned video highlight segment recognition model.

[0119] Figure 4 This is a flowchart illustrating a verification method for a video highlight segment recognition model provided in an embodiment of this specification.

[0120] From a programming perspective, the entity executing the process can be a program hosted on an application server or application terminal. It can be understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities.

[0121] like Figure 4 As shown, the process may include the following steps: Step 302: Obtain the baseline video-level prediction score corresponding to the complete segment sequence; Step 304: Set the input features of each video segment to zero in sequence to obtain the occlusion prediction score of each video segment after occlusion; Step 306: Use the difference between the baseline video-level prediction score and each occlusion prediction score as the leave-one contribution for the corresponding video segment; Step 308: Verify the highlight segment recognition result based on the ranking correlation between the leave-one contribution of each video segment and the highlight score of each video segment, and / or, based on the overlap rate between the top K segments with the leave-one contribution and the top K segments with the highlight score, where K is a positive integer.

[0122] In some embodiments, since only video-level business metrics are used as supervision signals during training, after model training, there are no corresponding real highlight labels for each video segment available for direct verification. To address this, a leave-one-out method can be used to verify the internal consistency of the model's highlight segment recognition results.

[0123] Specifically, for a test video to be validated, the baseline video-level prediction score b corresponding to its complete segment sequence is first obtained. This baseline score can be a multi-instance aggregated score m calculated by the model based on all valid video segments, reflecting the model's prediction of the overall business performance of the video.

[0124] Then, the input features of each video segment are sequentially set to zero (i.e., the segment is masked). This includes setting all inputs such as visual features, audio features, character features, temporal numerical features, and segment duration to zero, while simultaneously invalidating the mask corresponding to the segment. The masking prediction score b_i after the segment is masked is obtained. This masking score reflects the model's prediction of the overall business performance of the video in the absence of information about the i-th segment.

[0125] The difference between the baseline video-level prediction score and the prediction scores of each occluded segment is taken as the leave-one contribution Δ_i = b - b_i for the corresponding video segment. This leave-one contribution quantifies the marginal impact of the i-th segment on the overall prediction of the model—if the video-level prediction score drops significantly after a segment is occluded (i.e., Δ_i is a large positive value), it indicates that the segment has a large positive contribution to the model prediction; if the prediction score increases after occlusion (i.e., Δ_i is a negative value), it indicates that the segment may have a negative impact on the overall performance.

[0126] After obtaining the leave-one contribution for each segment, the highlight segment recognition results can be verified by the ranking correlation between the leave-one contribution of each segment and the highlight score output by the model. For example, the Spearman ranking correlation coefficient can be calculated: if segments with higher highlight scores generally have higher leave-one contributions, it indicates that the highlight ranking output by the model is consistent with the actual impact of each segment on video-level prediction. Furthermore, the overlap rate between the top K segments with the highest leave-one contribution and the top K segments with the highest highlight scores can be calculated, where K is a positive integer (e.g., 3 or 5). This metric measures whether the top K highlight segments selected by the model are indeed the top K segments with the highest leave-one contribution. These two verification metrics can quantitatively evaluate the highlight recognition results from two different dimensions: "global ranking rationality" and "head hit accuracy"—ranking correlation focuses on whether the overall order of all segments is reasonable, while Top-K overlap rate focuses on whether the selected segments were correctly identified.

[0127] In some embodiments, model performance is evaluated on a validation set (e.g., a video-level validation set containing 616 content samples). After model training, the Pearson correlation coefficient between the obtained click-through rate predictions and the actual smoothed click-through rates reaches 0.4995, with a root mean square error of 0.00821 and a mean absolute error of 0.00384. After sorting by model predictions, the actual average click-through rate of the top 10% of samples is 0.01613, and the actual average click-through rate of the bottom 10% of samples is 0.00375, corresponding to an improvement factor of approximately 4.31. Leave-one-out validation is performed on validation videos containing at least two valid segments (e.g., 596 videos). The average Spearman correlation coefficient between the duration-normalized score and the leave-one-out contribution reaches 0.6061, and the average overlap rate of the top-3 segments reaches 0.7081. The average leave-one-out contribution of the duration-normalized top-3 segments is approximately 0.04274, while the average leave-one-out contribution of the bottom-3 segments is approximately -0.01225. The experimental values ​​mentioned above are only used to illustrate the feasibility of one implementation and do not constitute a limitation on the data scale, parameter values, or effect threshold.

[0128] Based on some embodiments of this specification, in the absence of segment-level human-made real labels, the leave-one-out verification mechanism—which sequentially masks each segment and observes the change in video-level prediction scores—can quantitatively assess the marginal impact of each segment on the model's prediction results. This impact (leave-one-out contribution) can serve as a proxy measure of the segment's true importance. By calculating the ranking correlation (measuring the overall ranking consistency of all segments) and Top-K overlap rate (measuring the accuracy of hitting the head segment) between the leave-one-out contribution and the model's output highlight score, the highlight recognition results can be quantitatively verified from different dimensions, providing an objective means for evaluating the model's credibility without manual annotation.

[0129] Based on the same idea, this specification also provides a video highlight segment recognition method corresponding to the training method of the above-mentioned video highlight segment recognition model.

[0130] Figure 5 This is a flowchart illustrating a video highlight segment recognition method provided in an embodiment of this specification.

[0131] From a programming perspective, the entity executing the process can be a program hosted on an application server or application terminal. It can be understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities.

[0132] like Figure 5 As shown, the process may include the following steps: Step 402: Obtain the target video to be identified.

[0133] In some embodiments, when highlight segment recognition is required, a target video to be identified is acquired. This target video can be any advertisement video to be processed, such as an already delivered advertisement video obtained from an ad delivery system, original footage to be edited read from a media library, or a video file to be analyzed received through a user upload interface.

[0134] The method for acquiring the target video can be flexibly selected according to the actual application scenario. In some embodiments, the video file can be read directly from local storage or network storage; in some embodiments, video data can be obtained from a remote server via a video stream address (such as a URL); in some embodiments, the uploaded video file can also be received from the user's terminal device. The acquired target video can have various common video container formats, such as MP4, FLV, AVI, etc., and the video can be decoded in subsequent processing to extract the image and audio data of each frame.

[0135] It should be noted that in the training phase, step 202 acquires training sample videos with video-level business metrics; while in step 402 of this inference phase, the target video of the highlight segment to be identified is acquired. The acquisition methods of the video data can be the same for both, the difference is that step 202 also needs to acquire the video-level business metrics (such as exposure, clicks, etc.) corresponding to the training sample video, while this step only needs to acquire the target video itself, without acquiring the corresponding business metrics.

[0136] Step 404: Determine the multiple video segments contained in the target video.

[0137] In some embodiments, after acquiring a target video, multiple video segments contained within the target video are identified. This can be achieved by performing scene detection on the target video.

[0138] Specifically, based on the detection of shot transitions in the target video—when there is a significant change in the visual content between adjacent frames (such as a sudden change in the color histogram, an increase in optical flow exceeding a threshold, etc.), the location is marked as the scene boundary, thereby determining the initial video segment divided by scene.

[0139] Then, based on the duration of each initial video segment, adaptive processing is performed on the initial video segments to obtain multiple video segments. Adaptive processing can include merging, retaining, or secondary segmentation, specifically as follows: For initial video segments with a duration less than the minimum duration threshold (e.g., 1 second), they are merged with adjacent scenes, and the duration of the merged scene does not exceed the maximum duration threshold (e.g., 6 seconds); initial video segments with a duration between the minimum and maximum duration thresholds are retained; for initial video segments with a duration greater than the maximum duration threshold (e.g., 6 seconds), overlapping secondary segmentation is performed within that segment according to a preset window length (e.g., 4 seconds) and a preset step size (e.g., 2 seconds) to obtain multiple sub-segments. It should be noted that the above values ​​are only an exemplary parameter configuration and can be adjusted in different application scenarios.

[0140] In some embodiments, to facilitate batch tensor computation, a maximum number of segments can be preset (e.g., a maximum of 40 segments). When the actual number of video segments obtained after the above adaptive processing exceeds the maximum number of segments, the excess portion is truncated; when the actual number of video segments is less than the maximum number of segments, zero-padding is used to fill the gap to the maximum number of segments, and a valid segment mask is generated simultaneously.

[0141] In the embodiments of this specification, a "video segment" refers to multiple basic processing units arranged in chronological order after scene segmentation of the target video, including real video segments and filled segments. A "valid video segment" refers to a non-filled segment that actually exists in the video segment, has real feature data, and participates in the model calculation. It is marked by a valid segment mask—for example, a mask value of 1 is assigned to valid video segments, and a mask value of 0 is assigned to filled segments. In the processing of related embodiments, filled segments do not participate in calculations such as attention weight calculation and reward mean calculation.

[0142] Based on some embodiments of this specification, scene detection and segmentation of the target video according to the scene are performed to ensure that the segment boundaries correspond to the shot transitions or semantic changes in the video content. By adaptively merging, retaining, or re-segmenting the segments according to their duration, the duration of each segment is controlled within a reasonable range, avoiding semantic fragmentation caused by excessively short segments and information redundancy caused by excessively long segments. At the same time, by fixing the number of segments to a preset maximum value and generating effective segment masks, videos of different lengths can be input into the model in a unified tensor form for batch processing, providing a standardized data foundation for subsequent feature extraction and bi-branch inference.

[0143] Step 406: Determine the contextual representation of each video segment.

[0144] In some embodiments, after obtaining multiple video segments of the target video, determining the context representation of each video segment specifically includes: first, extracting input features for each video segment; the input features include at least one of the following: visual features, audio features, segment character features, and temporal numerical features.

[0145] Visual features can reflect the image content information of a video segment. In some embodiments, image features of each frame in a video segment can be extracted, and then average pooling or max pooling can be performed on the features of each frame within the segment to obtain the visual embedding of the segment. In one example, the visual embedding can be 2048-dimensional.

[0146] Audio features can reflect sound information such as background music and human voices in a video clip. In some embodiments, acoustic features of each audio frame in a video clip can be extracted, and then the features of each frame within the clip can be pooled to obtain the audio embedding of the clip. In one example, the audio embedding can have 1024 dimensions.

[0147] The segment role features are used to characterize the semantic role that the corresponding video segment plays in the video narrative structure of the target video. E-commerce advertising videos typically consist of multiple segments with different narrative functions, such as opening attraction, pain point description, product display, usage demonstration, selling point explanation, price promotion, trust verification, comparison display, action guidance, brand identification, and transitions. Different advertising videos may contain some or all of the above role types. The segment role features can be probability vectors, where each element corresponds to a preset role label (e.g., opening attraction, pain point description, product display, etc.). The values ​​of each element are independent, and each element's value is between 0 and 1, representing the probability that the video segment belongs to the corresponding preset role label. The probabilities of each role are independent and Softmax normalization is not performed, which allows a segment to have multiple role attributes simultaneously—for example, a segment can be both a "usage demonstration" and a "selling point explanation," or it can contain both "brand identification" and "action guidance."

[0148] The probabilities of characters in a video clip can be generated in various ways. In some embodiments, the visuals of the video clip, the speech-recognized text, and the optical character recognition text can be input into a multimodal large model (e.g., a visual-language model), which then outputs the independent probabilities of each preset character label. In other embodiments, character recognition and probability generation can also be performed using methods based on manual rules or dedicated classifiers.

[0149] The temporal numerical features are used to characterize at least one of the position and duration of the corresponding video segment on the timeline of the target video. In some embodiments, the temporal numerical features can be represented as a four-dimensional vector, including the segment duration, the proportion of the segment's starting position to the total video duration, an identifier indicating whether it is a sub-segment obtained by secondary segmentation of a long scene, and the segment's start time. This temporal information enables the model to perceive the relative position and duration distribution of each segment on the overall timeline of the video.

[0150] In some embodiments, after extracting the input features of each video segment, the input features of each video segment are input into the encoding network of the pre-trained model to obtain the contextual representation of each video segment. The encoding network may include a segment encoder and a cross-segment encoder. The pre-trained model includes the encoding network and attention and reward branches connected to the encoding network.

[0151] First, the input features of each video segment are input into a segment encoder to obtain the segment latent representation of each video segment. In some embodiments, visual embeddings and audio embeddings are projected onto a preset latent space through linear layers; temporal numerical features are mapped to a lower-dimensional numerical feature space through linear layers; and the character probability vector is multiplied by a trainable character embedding matrix to obtain the character embedding. Then, the projected visual features, audio features, temporal numerical features, and character embeddings are concatenated, and the concatenated features are input into a segment multilayer perceptron to obtain the initial latent representation of each segment. This segment multilayer perceptron may sequentially include a first linear layer, a layer normalization layer, a GELU activation layer, a Dropout layer, a second linear layer, and a second layer normalization layer. In one example, the latent dimension can be set to 256 dimensions, and the Dropout ratio can be set to 0.1.

[0152] Then, the hidden representations of each video segment are input into the cross-segment encoder to obtain the context representation of each video segment. The cross-segment encoder can adopt a Transformer encoder structure, which can model the temporal dependencies and semantic cooperation relationships between segments. The cross-segment encoder can use an effective segment mask as a key padding mask to avoid padding positions participating in self-attention computation and context aggregation. In one example, the cross-segment encoder can adopt a 1-layer Transformer encoder, 4 attention heads, and a 1024-dimensional feedforward network.

[0153] Based on some embodiments of this specification, by extracting multi-dimensional input features such as visual, audio, role, and temporal information, the model can simultaneously acquire content information (images and sounds), semantic information (role function in the advertising narrative), and structural information (position and duration on the video timeline) of a segment, providing a rich information foundation for subsequent segment representation learning. In particular, the role features adopt the form of mutually independent probability vectors, enabling a segment to simultaneously assume multiple semantic roles, which is more in line with the actual situation of overlapping segment functions in e-commerce advertising videos. By fusing multimodal features into segment latent representations through a segment encoder, and then modeling the contextual relationships between segments through a cross-segment Transformer encoder, the representation of each segment not only contains its own content information, but also incorporates temporal associations and semantic coordination information with preceding and following segments, thereby improving the accuracy and contextual consistency of segment representation.

[0154] Based on some embodiments of this specification, by sequentially inputting the input features of each video segment into a segment encoder and a cross-segment encoder, the segment encoder fuses multimodal features into a segment-level latent representation, and the cross-segment encoder (such as Transformer) models the global temporal dependencies between segments through a self-attention mechanism, so that the context representation of each segment not only contains its own content information, but also incorporates semantic associations and temporal collaboration information with other segments in the video, thereby providing a more accurate and context-consistent segment representation for subsequent attention and reward branches.

[0155] Step 408: Input the context representation of each video segment into the attention branch of the pre-trained model to obtain the attention weight of each video segment; the attention weight is used to characterize the aggregate weight of the corresponding video segment when generating the overall representation of the target video.

[0156] In some embodiments, after obtaining the context representation of each video segment, the context representation of each video segment is input into the attention branch of the pre-trained model to obtain the attention weight of each video segment. This attention weight is used to characterize the aggregate weight of the corresponding video segment in generating the overall representation of the target video.

[0157] In the embodiments of this specification, a "branch" refers to a functional division within the same neural network model, emphasizing the characteristics of shared underlying structure and joint training. A branch is a part that branches off from an intermediate layer in a neural network model and performs a specific sub-function. Different branches share the encoding network, but each has independent output heads and parameters to complete different prediction or computation tasks. In this embodiment, the attention branch and the reward branch (see step 410) share the encoding network of the pre-trained model, outputting attention weights and segment reward values, respectively.

[0158] The attention branch maps the context representation of each video segment to a scalar raw attention value. This raw attention value characterizes the unnormalized significance of the corresponding segment in the temporal context. The raw attention values ​​are then normalized (e.g., using the Softmax function) to obtain the attention weights for each video segment. The attention weights satisfy the condition that the sum of the weights of all valid video segments is 1, giving each segment's attention weight a clear probabilistic meaning. During the normalization process, the raw attention value for filler segments can be set to a minimum, causing the attention weights obtained by filler segments after normalization to approach zero, without affecting the weight distribution of valid segments.

[0159] The physical meaning of the aforementioned attention weights is limited to the numerical proportion of the representation segment during feature aggregation, that is, the weight coefficient of the segment when generating the overall representation of the target video by weighted summation of the features of each segment. Attention weights do not determine the positive or negative contribution of a segment—that function is handled independently by the reward branch.

[0160] Based on some embodiments of this specification, attention weights are output by the attention branch to characterize the numerical proportion of segments during feature aggregation. These weights are only used to determine the contribution ratio of each segment when generating the overall representation of the video, and do not involve the judgment of the contribution direction (positive or negative) of the segments. At the same time, the attention branch and the reward branch share the encoding network but output different physical quantities, so that the two branches are functionally independent and do not interfere with each other. The attention branch only needs to focus on the representativeness of the segment features, without having to consider the judgment of business contribution, thereby reducing the learning burden of a single branch and providing a foundation for the subsequent dual-branch decoupling mechanism.

[0161] Step 410: Input the context representation of each video segment into the reward branch of the pre-trained model to obtain the segment reward value of each video segment; the segment reward value is used to characterize the degree of business contribution of the corresponding video segment to the target video.

[0162] In the embodiments of this specification, the model outputs two distinct quantities for each segment: an attention weight and a segment reward value. The attention weight describes the degree to which the segment participates in the overall video representation aggregation (i.e., the numerical proportion of the segment in the feature aggregation), while the segment reward value describes the potential contribution of the segment relative to the video-level objective (i.e., the degree of positive or negative impact of the segment on business performance).

[0163] Specifically, attention weights are only used to characterize the numerical proportion of fragments when they are aggregated in the feature space, and do not serve to determine positive or negative contributions; fragment reward values ​​are specifically used to characterize the degree of contribution of fragments to business objectives. The output of the reward branch does not affect the calculation of attention weights, nor does the attention branch directly participate in the determination of business contributions. Through this design, the attention branch only needs to focus on the representativeness of fragment features, and the reward branch only needs to focus on the relevance of fragments to business objectives; the two do not interfere with each other.

[0164] Based on some embodiments of this specification, by outputting attention weights and fragment reward values ​​from two independent branches and clearly distinguishing them in physical meaning—attention weights only represent the numerical proportion of fragments during feature aggregation, while fragment reward values ​​specifically represent the degree of contribution of fragments to business objectives—the model avoids gradient conflicts caused by a single score simultaneously serving as both aggregation weight and contribution criterion during learning. The two branches learn independently and do not interfere with each other; the attention branch only needs to focus on the representativeness of fragment features, and the reward branch only needs to focus on the relevance of fragments to business objectives, thereby improving the stability of training and the interpretability of the model.

[0165] In one or more embodiments of this specification, the step of inputting the context representation of each video segment into the attention branch of a pre-trained model to obtain the attention weight of each video segment includes: the attention branch mapping the context representation of each video segment to the original attention value; and normalizing the original attention value using a temperature scaling parameter to obtain the attention weight of each video segment.

[0166] In some embodiments, the attention branch maps the context representation of each video segment to raw attention values. Specifically, the attention branch may use a single linear layer to map the context representation zi to a scalar ui, which represents the unnormalized significance of the corresponding segment in the temporal context. Then, a temperature scaling parameter is used to normalize the raw attention values ​​to obtain the attention weights for each video segment.

[0167] For example, attention weights can be calculated using a Softmax function with a temperature parameter: ai = exp(ui / τ) / Σj exp(uj / τ); Where ai is the attention weight of the i-th video segment, ui is the original attention value of the i-th video segment, and τ is the temperature scaling parameter. ui is non-linearly amplified using the exponential function exp and divided by the sum of the exponents of all valid segments, mapping ui to the (0,1) interval, while ensuring that the sum of the attention weights of all valid segments equals 1. The summation is only applied to valid video segments. Before normalization, the original attention value of the filler segment can be set to the minimum value that the data type can represent, so that the attention weight obtained by the filler segment after normalization approaches zero.

[0168] In some embodiments, the temperature scaling parameter τ can be set as a learnable parameter and constrained during training to be no less than a preset positive number (e.g., 0.1). When τ is large, the attention weights of each segment tend to be evenly distributed; when τ is small, the attention weights tend to be one-hot distributed (i.e., concentrated on individual segments). The temperature scaling parameter can also be equivalently defined as the inverse temperature form, i.e., normalized using ui × β (where β = 1 / τ).

[0169] Based on some embodiments of this specification, by normalizing the original attention values ​​using a learnable temperature scaling parameter, the model can adaptively adjust the concentration of attention distribution according to different video content during training. For example, for video clip sequences with a fast pace and concentrated climax, the model can make the attention distribution sharper to focus on key segments; for videos with smooth content and multiple segments of similar importance, the model can make the attention distribution smoother to evenly focus on each segment. At the same time, by imposing a constraint on the temperature parameter not lower than a preset positive number, the model avoids numerical instability and overly sharp attention distribution caused by the temperature approaching zero, so that the allocation of attention weights can be adapted to the content characteristics of different videos.

[0170] In one or more embodiments of this specification, the step of inputting the context representation of each video segment into the reward branch of the pre-trained model to obtain the segment reward value of each video segment includes: the reward branch mapping the context representation of each video segment to the segment reward value in an unbounded real number space.

[0171] Specifically, the reward branch can be implemented using a two-layer perceptron, mapping the context representation to a segment reward value ri without probability compression (e.g., without the sigmoid function), meaning that the value of ri can be any real number in the range (-∞, +∞). The weights of the reward output layer can be initialized with a small gain, and the bias can be initialized to zero to reduce numerical instability in the early stages of training.

[0172] The segment reward value is used to characterize the degree of business contribution of the corresponding video segment to the target video—a positive value indicates that the segment has a positive contribution to the business objective, and a negative value indicates that the segment has a negative contribution to the business objective. The magnitude of the absolute value reflects the degree of contribution. Since the reward value is in an unbounded real number space, the reward values ​​of different segments can show significant numerical differentiation, thereby maintaining the distinguishability between segments.

[0173] Based on some embodiments of this specification, the context representation is directly mapped to the fragment reward value in the unbounded real space through reward branching without probability compression, so that the fragment reward value can freely take values ​​in the range of (-∞, +∞) - fragments with larger positive contributions can take larger positive values, and fragments with smaller or negative contributions can take smaller or negative values ​​- thereby maintaining the distinguishability between different fragments in terms of numerical value, and providing a distinguishable reward signal for subsequent weighted aggregation and highlight score calculation.

[0174] Step 412: The attention weight and segment reward value of each video segment are fused to obtain the highlight score of each video segment. The highlight score increases with the increase of the attention weight and the increase of the segment reward value.

[0175] In some embodiments, after obtaining the attention weight a_i and segment reward value r_i for each video segment, the two are fused to obtain the highlight score for each video segment. This highlight score is used to measure the overall score of each segment after comprehensively considering its contextual importance and business contribution, and thus serves as the basis for subsequently determining highlight segments.

[0176] In one implementation, the fusion process can be in multiplicative form: h_i = a_i × r_i, where h_i is the highlight score of the i-th video segment. In this approach, the highlight score simultaneously considers the aggregation weight (a_i) of the segment in the overall video representation and the segment's contribution to the business objective (r_i). That is, calculating the highlight score of each video segment includes multiplying the attention weight of each video segment by the segment reward value to obtain the highlight score of each video segment.

[0177] In some embodiments, when fusion is performed using the multiplicative form h_i = a_i × r_i, since the segment reward value r_i is in an unbounded real space, the reward values ​​of different segments can be sufficiently differentiated numerically. Even if the overall performance of a video is poor, resulting in negative r_i values ​​for all segments, the absolute values ​​of each segment still differ, and the ranking of h_i can still reflect the relative merits of each segment, thus maintaining the discriminative power of highlight determination. Based on this, a video-internal reward centering mechanism can be used to obtain relatively positive reward values ​​mapped to the (0,1) interval, further improving the interpretability of highlight scores.

[0178] In another implementation, the fusion process can be additive: h_i = a_i + r_i. In this method, the attention weights and fragment reward values ​​need to be normalized to the same dimension before addition. In yet another implementation, the fusion process can be a weighted sum: h_i = α × a_i + β × r_i, where α and β are preset coefficients used to adjust the relative weights of attention weights and fragment reward values ​​in the highlight score.

[0179] Regardless of the fusion method used, the highlight score should increase with the increase of the attention weight and the segment reward value—that is, the highlight score should have a monotonically increasing relationship with both the attention weight and the segment reward value. This ensures that the model assigns higher scores to segments that are "highly noticed and highly contributing." Specifically, the fusion process of the attention weight and segment reward value for each video segment includes: calculating the highlight score for each video segment based on its attention weight and segment reward value. The highlight score increases with both the attention weight and the segment reward value. All of the above fusion methods can achieve the technical objective of comprehensively considering the attention weight and segment reward value to determine highlight segments.

[0180] Based on some embodiments of this specification, a highlight score is obtained by fusing attention weights and segment reward values. Regardless of whether multiplication, addition, or weighted summation is used, the highlight score increases with the increase of attention weights and segment reward values. This means that highlight segments need to simultaneously meet the two conditions of "having a high aggregation weight in the overall video representation" and "having a high degree of contribution to business objectives." Compared with using attention weights or segment reward values ​​alone, this method can more comprehensively measure the value of segments.

[0181] In one or more embodiments of this specification, the step of fusing the attention weights and segment reward values ​​of each video segment to obtain the highlight score of each video segment includes: calculating the average reward value based on the segment reward values ​​of each video segment within the same target video; determining the relative positive reward value of each video segment based on the difference between the segment reward value of each video segment and the average reward value; and determining the highlight score of each video segment based on the attention weights and the relative positive reward values ​​of each video segment.

[0182] In some embodiments, the fusion process employs an in-video reward centralization mechanism. First, the average reward is calculated based on the segment reward values ​​of each video segment within the same target video. The multiple video segments include valid video segments and filler video segments; filler video segments are not included in the calculation of the average reward. Valid video segments can be determined from the multiple video segments based on a valid segment mask—the valid segment mask assigns a first mask value (e.g., 1) to valid video segments and a second mask value (e.g., 0) to filler video segments. Then, the average reward is calculated only based on the segment reward values ​​of the valid video segments. .

[0183] Then, based on the difference between the segment reward value and the average reward value of each video segment, the relative positive reward value of each video segment is determined. Specifically, for the i-th valid video segment, its relative positive reward value can be expressed as q_i = sigmoid(r_i - The relative positive reward value is obtained by subtracting the video mean from the segment reward value (centering), and then mapping it to the (0,1) interval using the Sigmoid function. This relative positive reward value reflects the contribution level of the segment relative to other segments in the same video—when the segment reward value is higher than the video mean, q_i is greater than 0.5, indicating that the segment's contribution is above average; when the segment reward value is lower than the video mean, q_i is less than 0.5, indicating that the segment's contribution is below average. Since the relative positive reward value revolves around 0.5, even if the click-through rate of the entire video is low, resulting in negative absolute reward values ​​for all segments, a distinctive relative positive reward value can still be generated based on the segment's level relative to the video mean.

[0184] Finally, the highlight score of each video segment is determined based on its attention weight and relative positive reward value. Specifically, the highlight score of each video segment can be obtained by multiplying its attention weight by its relative positive reward value, resulting in the score h_i = a_i × q_i.

[0185] Based on some embodiments of this specification, a video reward centralization mechanism is used to first calculate the average reward of valid segments within the same video. Then, the reward value of each segment is subtracted from the average value, and a relative positive reward value is obtained through Sigmoid mapping. This relative positive reward value reflects the contribution level of a segment relative to other segments within the same video, rather than the absolute contribution size. This ensures that even if the overall performance of the video is poor and the absolute reward values ​​of all segments are negative, a relative positive reward value distributed around 0.5 can still be generated based on the segment's level relative to the average value within the video. This allows for the identification of relatively high-quality segments even in videos with low performance, avoiding the problem that all segments cannot obtain highlight scores due to poor overall video performance.

[0186] In one or more embodiments of this specification, determining the relative positive reward value of each video segment based on the difference between the segment reward value of each video segment and the average reward value includes: calculating the difference between the segment reward value of each video segment and the average reward value; inputting the difference corresponding to each video segment into a probability compression function, and obtaining the output of the probability compression function as the relative positive reward value of each video segment.

[0187] In some embodiments, the specific method for determining the relative positive reward value is as follows: First, calculate the difference between the segment reward value and the average reward value of each video segment (i.e., r_i - Then, the difference between each video segment is input into the probability compression function, and the output of the probability compression function is used as the relative positive reward value of each video segment.

[0188] The probability compression function is a monotonically increasing function used to map any real number to the interval (0,1).

[0189] In some embodiments, the probability compression function can be the Sigmoid function, i.e., q_i = sigmoid(r_i- The Sigmoid function maps the difference to the (0,1) interval—when the difference is positive, the output is greater than 0.5, indicating that the contribution of the segment is higher than the average level in the video; when the difference is negative, the output is less than 0.5, indicating that the contribution of the segment is lower than the average level in the video.

[0190] In other examples, the probability compression function can also be other mapping functions with monotonically increasing properties, as long as they can map the input values ​​to the (0,1) interval. For example, a translational scaling form of the hyperbolic tangent function can be used: q_i = (tanh(r_i - The function `(0,1) + 1) / 2` also maps the difference to the (0,1) interval, preserving its monotonically increasing and centrally symmetric properties. Alternatively, a hard Sigmoid function can be used, which approximates the Sigmoid function in a piecewise linear manner: outputting 0 when the difference is less than a preset lower bound and outputting 1 when the difference is greater than a preset upper bound, with linear interpolation between the preset lower and upper bounds. This function maintains its monotonically increasing property while having lower computational overhead. Although the specific mapping curves of these different implementations of probability compression functions differ, they all satisfy the core constraint of monotonically increasing the mapping of the input from the unbounded real number space to the (0,1) interval, ensuring that the relative positive reward values ​​of each video segment reflect the relative magnitude relationship between the segment reward values.

[0191] Because the probability compression function has a monotonically increasing property, its output value retains the relative ranking relationship between the reward values ​​of the segments—segments with larger differences receive higher relative positive reward values, while segments with smaller differences receive lower relative positive reward values.

[0192] Based on some embodiments of this specification, by mapping the difference between the fragment reward value and the mean reward value to the (0,1) interval through a probability compression function (e.g., Sigmoid), the relatively positive reward value has a clear probabilistic meaning—greater than 0.5 indicates a contribution above average, and less than 0.5 indicates a contribution below average—while preserving the relative ranking relationship between fragments, and uniformly mapping the uncompressed fragment reward value (which may be in a large or small numerical range) to a fixed interval, providing a dimensionally consistent basis for subsequent multiplication with attention weights (also in the (0,1) interval).

[0193] Step 414: Determine the highlight segments from the video segments based on the highlight scores of each video segment.

[0194] Specifically, video clips can be sorted from highest to lowest highlight score, and one or more of the top-ranked clips can be identified as highlight clips.

[0195] In some embodiments, before determining highlight segments, the highlight scores can be normalized based on the duration of each video segment; highlight segments are then determined from the video segments based on the normalized scores. Duration normalization can be performed as follows: for the i-th video segment, calculate its normalized score s_i = h_i / √d_i, where h_i is the highlight score of the segment, and d_i is the duration of the segment (a positive lower limit can be set to avoid the denominator being zero). This normalization method can reduce the bias of longer segments naturally receiving higher scores due to their greater content content—since the highlight score is obtained by multiplying the attention weight by the segment reward value, longer segments may naturally have higher cumulative scores. Dividing by the square root of the duration can weaken the influence of duration on the score, allowing segments with different durations but similar levels of brilliance to obtain similar normalized scores.

[0196] In some embodiments, after identifying the highlight segments, post-processing can be performed on the candidate highlight segments. Specifically, candidate highlight segments are first identified from each video segment based on their highlight scores (or duration-normalized scores). Then, candidate highlight segments that are temporally adjacent and have an interval less than a preset merging threshold are merged into a single target highlight segment. For example, if two candidate highlight segments have an interval of less than 0.5 seconds on the timeline, they can be considered to belong to the same continuous segment of exciting content, and they can be merged and output as a continuous time interval. After merging, information such as the start and end times, main character tags, attention weights, original reward values, highlight scores, and duration-normalized scores of the target highlight segment can be output.

[0197] Based on some embodiments of this specification, highlight scores are corrected by duration normalization, eliminating the impact of segment duration differences on score ranking. This ensures that shorter but more engaging segments are not underestimated due to accumulated numerical disadvantages, thus improving the fairness of highlight ranking. By merging candidate segments with adjacent times, the same continuous and engaging content is prevented from being split into multiple fragmented segments, resulting in output highlight segments with more complete time intervals, which is convenient for subsequent applications such as material editing or content summarization.

[0198] Based on the same idea, embodiments of this specification also provide apparatus corresponding to the above methods.

[0199] Figure 6 The embodiments provided in this specification correspond to Figure 3 A schematic diagram of the structure of a training device for a video highlight segment recognition model.

[0200] like Figure 6 As shown, the device may include: The training sample and metric acquisition module 502 is configured to acquire training sample videos and video-level business metrics corresponding to the training sample videos. The video segment segmentation module 504 is configured to determine multiple video segments contained in the training sample video; Context representation module 506 is configured to determine the context representation of each video segment; The attention weight prediction module 508 is configured to input the context representation of each video segment into the attention branch of the model to be trained to obtain the attention weight of each video segment; the attention weight is used to characterize the aggregate weight of the corresponding video segment when generating the overall representation of the training sample video. The segment reward value prediction module 510 is configured to input the context representation of each video segment into the reward branch of the model to be trained to obtain the segment reward value of each video segment; the segment reward value is used to characterize the degree of contribution of each video segment to the video-level business indicators. The aggregation module 512 is configured to perform weighted aggregation of the segment reward values ​​of each video segment based on the attention weight of each video segment to obtain the aggregation result; The parameter update module 514 is configured to update the network parameters of the model to be trained based on the difference between the aggregation result and the video-level service metric.

[0201] based on Figure 6 The embodiments of this specification also provide some specific implementation schemes of the method, which are described below.

[0202] Optionally, determining the context representation of each video segment includes: Extract the input features of each video segment; The input features of each video segment are input into the encoding network of the model to be trained to obtain the contextual representation of each video segment.

[0203] Optionally, the context representation of each video segment is determined based on at least one of the following features: visual features, audio features, segment character features, and temporal numerical features; The segment role feature is used to characterize the semantic role that the corresponding video segment plays in the video narrative structure of the training sample video, and the temporal numerical feature is used to characterize at least one of the position and duration of the corresponding video segment on the timeline of the training sample video.

[0204] Optionally, the step of inputting the context representation of each video segment into the attention branch of the model to be trained to obtain the attention weights of each video segment includes: The attention branch maps the context representation of each video segment to the original attention value; The original attention values ​​are normalized using a temperature scaling parameter to obtain the attention weights for each video segment.

[0205] Optionally, the step of inputting the context representation of each video segment into the reward branch of the model to be trained to obtain the segment reward value of each video segment includes: The reward branch maps the context representation of each video segment to a segment reward value in an unbounded real space; The step of updating the network parameters of the model to be trained based on the difference between the aggregation result and the video-level service metric includes: The aggregation loss is calculated based on the difference between the aggregation result and the logarithmic probability transformation value of the video-level business metric. The network parameters of the model to be trained are updated based on the aggregate loss.

[0206] Optionally, the video-level business metric includes smoothed click-through rate (CTR); the smoothed CTR is calculated based on the exposure, clicks, and global CTR of the training sample videos.

[0207] Optionally, it also includes: Based on the aggregation results, predict auxiliary business metrics for the training sample videos; the auxiliary business metrics include at least one of the following: cost per thousand impressions, cost per click, and relative global click-through rate improvement; Calculate the auxiliary loss between the predicted and actual values ​​of the auxiliary business indicators; The network parameters of the model to be trained are updated based on the auxiliary loss.

[0208] Optionally, it also includes: Calculate the variance of the reward value for each video segment; When the variance is lower than a preset minimum variance threshold, the reward dispersion loss is calculated based on the difference between the variance and the preset minimum variance threshold. The network parameters of the model to be trained are updated based on the reward dispersion loss.

[0209] Optionally, it also includes: The entropy of the attention weight is normalized based on the number of valid video segments among the plurality of video segments; When the normalized entropy is lower than the preset lower limit, the attention entropy constraint loss is calculated based on the difference between the normalized entropy and the preset lower limit. The network parameters of the attention branch are updated based on the attention entropy constraint loss.

[0210] Optionally, it also includes: Calculate the difference in attention weights between temporally adjacent valid video segments among the plurality of video segments; Calculate the adjacent attention smoothing loss based on the aforementioned differences; The network parameters of the attention branch are updated based on the adjacent attention smoothing loss.

[0211] It is understood that the modules mentioned above refer to computer programs or program segments used to perform one or more specific functions. Furthermore, the distinction between these modules does not imply that the actual program code must also be separate.

[0212] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0213] The above is a schematic scheme of a training device for a video highlight segment recognition model according to this embodiment. It should be noted that the technical solution of this training device for the video highlight segment recognition model and the technical solution of the training method for the video highlight segment recognition model described above belong to the same concept. For details not described in detail in the technical solution of the training device for the video highlight segment recognition model, please refer to the description of the technical solution of the training method for the video highlight segment recognition model described above.

[0214] Based on the same idea, embodiments of this specification also provide apparatus corresponding to the above methods.

[0215] Figure 7 The embodiments provided in this specification correspond to Figure 5 A schematic diagram of the structure of a video highlight segment recognition device.

[0216] like Figure 7 As shown, the device may include: The target video acquisition module 602 is configured to acquire the target video to be identified. The video segment determination module 604 is configured to determine multiple video segments contained in the target video; The context representation determination module 606 is configured to determine the context representation of each video segment; The attention weight determination module 608 is configured to input the context representation of each video segment into the attention branch of the pre-trained model to obtain the attention weight of each video segment; the attention weight is used to characterize the aggregate weight of the corresponding video segment when generating the overall representation of the target video. The segment reward value determination module 610 is configured to input the context representation of each video segment into the reward branch of the pre-trained model to obtain the segment reward value of each video segment; the segment reward value is used to characterize the degree of business contribution of the corresponding video segment to the target video; The highlight score calculation module 612 is configured to fuse the attention weight and the segment reward value of each video segment to obtain the highlight score of each video segment; the highlight score increases with the increase of the attention weight and the increase of the segment reward value. The highlight segment determination module 614 is configured to determine highlight segments from each video segment based on the highlight score of each video segment.

[0217] based on Figure 7 The embodiments of this specification also provide some specific implementation schemes of the method, which are described below.

[0218] Optionally, determining the context representation of each video segment includes: Extract the input features of each video segment; The input features of each video segment are input into the encoding network of the pre-trained model to obtain the contextual representation of each video segment.

[0219] Optionally, the context representation of each video segment is determined based on at least one of the following features: visual features, audio features, segment character features, and temporal numerical features; The segment role feature is used to characterize the semantic role that the corresponding video segment plays in the video narrative structure of the target video, and the temporal numerical feature is used to characterize at least one of the position and duration of the corresponding video segment on the timeline of the target video.

[0220] Optionally, the step of inputting the input features of each video segment into the encoding network of the pre-trained model to obtain the contextual representation of each video segment includes: The input features of each video segment are input into the segment encoder to obtain the segment latent representation of each video segment; The segment implicit representation of each video segment is input into the cross-segment encoder to obtain the context representation of each video segment.

[0221] Optionally, the step of inputting the context representation of each video segment into the attention branch of the pre-trained model to obtain the attention weights of each video segment includes: The attention branch maps the context representation of each video segment to the original attention value; The original attention values ​​are normalized using a temperature scaling parameter to obtain the attention weights for each video segment.

[0222] Optionally, the step of inputting the context representation of each video segment into the reward branch of the pre-trained model to obtain the segment reward value of each video segment includes: The reward branch maps the context representation of each video segment to a segment reward value in an unbounded real space.

[0223] Optionally, the process of fusing the attention weights and segment reward values ​​of each video segment to obtain the highlight score of each video segment includes: Calculate the average reward based on the segment reward values ​​of each video segment within the same target video; The relative positive reward value of each video segment is determined based on the difference between the segment reward value of each video segment and the average reward value. The highlight score of each video segment is determined based on the attention weight and the relative positive reward value of each video segment.

[0224] Optionally, determining the relative positive reward value of each video segment based on the difference between the segment reward value and the average reward value includes: Calculate the difference between the segment reward value of each video segment and the average reward value; The difference corresponding to each video segment is input into a probability compression function, and the output of the probability compression function is used as the relative positive reward value of each video segment.

[0225] Optionally, determining highlight segments from the video segments based on the highlight scores of each video segment includes: The highlight scores are normalized based on the duration of each video segment. Highlight segments are determined from each video segment based on the scores after duration normalization.

[0226] It is understood that the modules mentioned above refer to computer programs or program segments used to perform one or more specific functions. Furthermore, the distinction between these modules does not imply that the actual program code must also be separate.

[0227] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0228] The above is a schematic scheme of a video highlight segment recognition device according to this embodiment. It should be noted that the technical solution of this video highlight segment recognition device and the technical solution of the video highlight segment recognition method described above belong to the same concept. For details not described in detail in the technical solution of the video highlight segment recognition device, please refer to the description of the technical solution of the video highlight segment recognition method described above.

[0229] Based on the same idea, this specification also provides devices corresponding to the above methods in its embodiments.

[0230] Figure 8 A structural block diagram of a computing device 700 provided according to an embodiment of this specification is shown.

[0231] The computing device 700 includes: Memory 710 and processor 720; The memory 710 is used to store computer programs / instructions, and the processor 720 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 720, they implement the training method of the video highlight segment recognition model, the verification method of the video highlight segment recognition model, or the steps of the video highlight segment recognition method.

[0232] Specifically, the components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0233] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0234] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0235] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0236] The processor 720 executes the computer instructions to implement the training method of the video highlight segment recognition model, the verification method of the video highlight segment recognition model, or the steps of the video highlight segment recognition method.

[0237] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the training method, verification method, or technical solution of the video highlight segment recognition model described above. Details not described in detail in the technical solution of the computing device can be found in the descriptions of the training method, verification method, or technical solution of the video highlight segment recognition model described above.

[0238] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the video highlight segment recognition model training method, the video highlight segment recognition model verification method, or the video highlight segment recognition method as described above.

[0239] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the training method, verification method, or technical solution of the video highlight segment recognition model described above. Details not described in detail in the technical solution of the storage medium can be found in the descriptions of the training method, verification method, or technical solution of the video highlight segment recognition model described above.

[0240] An embodiment of this specification also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the training method of the video highlight segment recognition model, the verification method of the video highlight segment recognition model, or the steps of the video highlight segment recognition method described above.

[0241] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the training method for the video highlight segment recognition model, the verification method for the video highlight segment recognition model, or the technical solution of the video highlight segment recognition method described above. Details not described in detail in the technical solution of the computer program product can be found in the descriptions of the training method for the video highlight segment recognition model, the verification method for the video highlight segment recognition model, or the technical solution of the video highlight segment recognition method.

[0242] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the apparatus and device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The apparatus, device and method provided in the embodiments of this specification are corresponding to each other, and therefore the apparatus and device also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus and device will not be repeated here.

[0243] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0244] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0245] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0246] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0247] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0248] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0249] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0250] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0251] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0252] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0253] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0254] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital character versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0255] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0256] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for identifying highlight segments in a video, comprising: Obtain the target video to be identified; Identify the multiple video segments contained in the target video; Determine the contextual representation of each video segment; The context representation of each video segment is input into the attention branch of the pre-trained model to obtain the attention weight of each video segment; the attention weight is used to characterize the aggregate weight of the corresponding video segment when generating the overall representation of the target video; The context representation of each video segment is input into the reward branch of the pre-trained model to obtain the segment reward value of each video segment; the segment reward value is used to characterize the degree of business contribution of the corresponding video segment to the target video; The attention weight and the segment reward value of each video segment are fused to obtain the highlight score of each video segment; the highlight score increases with the increase of the attention weight and the increase of the segment reward value. Highlight segments are determined from each video segment based on the highlight score of each video segment.

2. The method according to claim 1, wherein determining the context representation of each video segment includes: Extract the input features of each video segment; The input features of each video segment are input into the encoding network of the pre-trained model to obtain the contextual representation of each video segment.

3. The method according to claim 1, wherein the context representation of each video segment is determined based on at least one of the following features: visual features, audio features, segment character features, and temporal numerical features; wherein The segment role feature is used to characterize the semantic role that the corresponding video segment plays in the video narrative structure of the target video, and the temporal numerical feature is used to characterize at least one of the position and duration of the corresponding video segment on the timeline of the target video.

4. The method according to claim 2, wherein inputting the input features of each video segment into the encoding network of the pre-trained model to obtain the context representation of each video segment includes: The input features of each video segment are input into the segment encoder to obtain the segment latent representation of each video segment; The segment implicit representation of each video segment is input into the cross-segment encoder to obtain the context representation of each video segment.

5. The method according to claim 1, wherein inputting the context representation of each video segment into the attention branch of the pre-trained model to obtain the attention weights of each video segment includes: The attention branch maps the context representation of each video segment to the original attention value; The original attention values ​​are normalized using a temperature scaling parameter to obtain the attention weights for each video segment.

6. The method according to claim 1, wherein inputting the context representation of each video segment into the reward branch of the pre-trained model to obtain the segment reward value of each video segment includes: The reward branch maps the context representation of each video segment to a segment reward value in an unbounded real space.

7. The method according to claim 1, wherein fusing the attention weights and segment reward values ​​of each video segment to obtain the highlight score of each video segment comprises: According to the segment reward values of the video segments in the same target video, a reward mean value is calculated; According to the difference between the segment reward value of each video segment and the reward mean value, a relative positive reward value of each video segment is determined; According to the attention weight of each video segment and the relative positive reward value, a highlight score of each video segment is determined.

8. The method of claim 7, wherein the relative positive reward value of each video segment is determined according to the difference between the segment reward value of each video segment and the reward mean value, comprising: calculating the difference between the segment reward value of each video segment and the reward mean value; inputting the difference corresponding to each video segment into a probability compression function to obtain the output of the probability compression function as the relative positive reward value of each video segment.

9. The method of claim 1, wherein the highlight segment is determined from the video segments according to the highlight score of each video segment, comprising: performing time length normalization on the highlight score according to the time length of each video segment; determining the highlight segment from the video segments according to the time length normalized score.

10. A training method of a video highlight segment identification model, comprising: obtaining a training sample video and a video-level service indicator corresponding to the training sample video; determining a plurality of video segments contained in the training sample video; determining a context representation of each video segment; inputting the context representation of each video segment into an attention branch of a to-be-trained model to obtain an attention weight of each video segment; the attention weight is used to represent the aggregation weight of the corresponding video segment in generating the overall representation of the training sample video; inputting the context representation of each video segment into a reward branch of the to-be-trained model to obtain a segment reward value of each video segment; the segment reward value is used to represent the contribution degree of each video segment to the video-level service indicator; performing weighted aggregation on the segment reward value of each video segment according to the attention weight of each video segment to obtain an aggregation result; updating the network parameters of the to-be-trained model according to the difference between the aggregation result and the video-level service indicator.

11. The method of claim 10, wherein the context representation of each video segment is determined, comprising: extracting an input feature of each video segment; inputting the input feature of each video segment into an encoding network of the to-be-trained model to obtain the context representation of each video segment.

12. The method of claim 10, wherein the context representation of each video segment is determined based on at least one of the following features: visual features, audio features, segment role features, and time sequence numerical features; wherein the segment role feature is used to represent the semantic role assumed by the corresponding video segment in the video narrative structure of the training sample video, and the time sequence numerical feature is used to represent at least one of the position and the continuous time length of the corresponding video segment on the time axis of the training sample video.

13. The method of claim 10, wherein the attention weight of each video segment is obtained by inputting the context representation of each video segment into the attention branch of the to-be-trained model, comprising: The attention branch maps the context representation of each video segment to an attention raw value; The attention raw value is normalized by a temperature scaling parameter to obtain an attention weight of each video segment.

14. The method of claim 10, wherein the inputting the context representation of each video segment into the reward branch of the to-be-trained model to obtain a segment reward value of each video segment comprises: The reward branch maps the context representation of each video segment to a segment reward value in an unbounded real number space; The updating the network parameters of the to-be-trained model according to the difference between the aggregation result and the video-level business indicator comprises: calculating an aggregation loss according to the difference between the aggregation result and a logit-transformed value of the video-level business indicator; updating the network parameters of the to-be-trained model according to the aggregation loss.

15. The method of claim 10, wherein the video-level business indicator comprises a smoothed click rate, and the smoothed click rate is calculated according to an exposure amount, a click amount and a global click rate of the training sample video.

16. The method of claim 15, further comprising: predicting an auxiliary business indicator of the training sample video according to the aggregation result, wherein the auxiliary business indicator comprises at least one of the following: a thousand-exposure consumption, a single-click cost, a relative global click rate improvement; calculating an auxiliary loss between a predicted value and an actual value of the auxiliary business indicator; updating the network parameters of the to-be-trained model according to the auxiliary loss.

17. The method of claim 10, further comprising: calculating a variance of the segment reward value of each video segment; when the variance is lower than a preset minimum variance threshold, calculating a reward dispersion loss according to the difference between the variance and the preset minimum variance threshold; updating the network parameters of the to-be-trained model according to the reward dispersion loss.

18. The method of claim 10, further comprising: normalizing an entropy of the attention weight according to the number of valid video segments in the plurality of video segments; when the normalized entropy is lower than a preset lower limit, calculating an attention entropy constraint loss according to the difference between the normalized entropy and the preset lower limit; updating the network parameters of the attention branch according to the attention entropy constraint loss.

19. The method of claim 10, further comprising: calculating a difference between the attention weights of time-adjacent valid video segments in the plurality of video segments; calculating an adjacent attention smoothing loss according to the difference; updating the network parameters of the attention branch according to the adjacent attention smoothing loss.

20. A verification method of a video highlight segment identification model, comprising: obtaining a reference video-level prediction score corresponding to a complete segment sequence; sequentially setting input features of each video segment to zero to obtain a masking prediction score of each video segment after being masked; taking a difference between the reference video-level prediction score and each masking prediction score as a leave-one-out contribution of the corresponding video segment. According to the ranking correlation between the leave-one-out contribution of each video segment and the highlight score of each video segment, and / or according to the coincidence rate between the top K segments ranked by the leave-one-out contribution and the top K segments ranked by the highlight score, the highlight segment recognition result is verified, where K is a positive integer.

21. A device for identifying a highlight segment of a video, comprising: a target video obtaining module configured to obtain a target video to be identified; a video segment determining module configured to determine a plurality of video segments contained in the target video; a context representation determining module configured to determine a context representation of each video segment; an attention weight determining module configured to input the context representation of each video segment into an attention branch of a pre-trained model to obtain an attention weight of each video segment; the attention weight is used to represent an aggregation weight of the corresponding video segment when generating an overall representation of the target video; a segment reward value determining module configured to input the context representation of each video segment into a reward branch of the pre-trained model to obtain a segment reward value of each video segment; the segment reward value is used to represent a service contribution degree of the corresponding video segment to the target video; a highlight score calculating module configured to perform fusion processing on the attention weight and the segment reward value of each video segment to obtain a highlight score of each video segment; the highlight score increases with the increase of the attention weight and increases with the increase of the segment reward value; a highlight segment determining module configured to determine a highlight segment from the plurality of video segments according to the highlight score of each video segment.

22. A training device of a model for identifying a highlight segment of a video, comprising: a training sample and index obtaining module configured to obtain a training sample video and a video-level service index corresponding to the training sample video; a video segment dividing module configured to determine a plurality of video segments contained in the training sample video; a context representation module configured to determine a context representation of each video segment; an attention weight predicting module configured to input the context representation of each video segment into an attention branch of a to-be-trained model to obtain an attention weight of each video segment; the attention weight is used to represent an aggregation weight of the corresponding video segment when generating an overall representation of the training sample video; a segment reward value predicting module configured to input the context representation of each video segment into a reward branch of the to-be-trained model to obtain a segment reward value of each video segment; the segment reward value is used to represent a contribution degree of each video segment to the video-level service index; an aggregation module configured to perform weighted aggregation on the segment reward value of each video segment according to the attention weight of each video segment to obtain an aggregation result; a parameter updating module configured to update network parameters of the to-be-trained model according to a difference between the aggregation result and the video-level service index.

23. A computing device, comprising: a memory and a processor; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the method according to any one of claims 1-20. 24.A computer readable storage medium storing computer instructions, wherein the computer instructions, when executed by a processor, implement the steps of the method according to any one of claims 1-20. 25.A computer program product comprising computer programs / instructions, wherein the computer programs / instructions, when executed by a processor, implement the steps of the method according to any one of claims 1-20.

Images