Key video segment determination method and video playing method

Abstract

The application relates to a key video segment determination method and a video playing method. The key video segment determination method comprises the following steps: acquiring a plurality of second videos related to first video content; determining a video segment matched with each of the plurality of second videos from the first video; determining a hotness value statistical unit in each video segment in the first video; acquiring a video hotness value of each of the plurality of second videos, and accumulating the video hotness value of each of the plurality of second videos to a unit hotness value of the hotness value statistical unit of each video segment; and determining a key video segment from the first video according to the unit hotness value of each hotness value statistical unit in the first video. The method can improve the positioning accuracy of the key video segment.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for determining key video segments, as well as a method, apparatus, computer device, computer-readable storage medium, and computer program product for playing videos. Background Technology

[0002] With the development of computer technology, key video clip identification technology has emerged. Key video clips refer to segments in a video that contain a large amount of information and can evoke discussion and resonance among viewers; they can also be called highlight segments. For example, in film and television dramas, key video clips often appear at crucial moments in the plot, such as comedic, conflict, or plot twist scenes, which are more likely to become key video clips.

[0003] In traditional techniques, the key video segments are determined by pre-training a supervised model based on manually labeled data, using multimodal techniques to calculate the features of each video frame in the video, using the features of each video frame to predict the starting video frame of the key video segment, and then taking video frames of a fixed length from the starting video frame as the key video segments.

[0004] However, traditional methods rely on manually labeled data during the pre-training of supervised models, which introduces a high degree of subjectivity and leads to inaccurate predictions of the starting video frame, resulting in low accuracy in locating key video segments. Summary of the Invention

[0005] Therefore, it is necessary to address the above-mentioned technical problems by providing a method, apparatus, computer device, computer-readable storage medium, and computer program product for determining key video segments that can improve the accuracy of key video segment positioning, as well as a video playback method, apparatus, computer device, computer-readable storage medium, and computer program product.

[0006] Firstly, this application provides a method for determining key video segments, including:

[0007] Retrieve multiple second videos related to the content of the first video;

[0008] From the first video, determine the video segments that match each of the plurality of second videos;

[0009] Determine the heat value statistical unit within each video segment of the first video;

[0010] Obtain the video popularity value of each of the plurality of second videos, and accumulate the video popularity values ​​of each of the plurality of second videos into the unit popularity value of the popularity value statistics unit of each of the video segments;

[0011] Based on the unit popularity value of each popularity value statistical unit in the first video, key video segments are determined from the first video.

[0012] Secondly, this application also provides a key video segment determination device, comprising:

[0013] The video acquisition module is used to acquire multiple second videos related to the content of the first video;

[0014] The segment matching module is used to determine, from the first video, video segments that match each of the plurality of second videos;

[0015] The statistical unit determination module is used to determine the heat value statistical unit in each video segment of the first video;

[0016] The popularity value accumulation module is used to obtain the video popularity value of each of the plurality of second videos and accumulate the video popularity values ​​of each of the plurality of second videos into the unit popularity value of the popularity value statistics unit of each video segment.

[0017] The segment determination module is used to determine key video segments from the first video based on the unit popularity value of each popularity value statistical unit in the first video.

[0018] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned key video segment determination method.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for determining key video segments.

[0020] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for determining key video segments.

[0021] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for determining key video segments acquire multiple second videos related to the content of a first video. From the first video, they determine video segments that match each of the multiple second videos, identify heat value statistical units within each video segment of the first video, obtain the video heat values ​​of each of the multiple second videos, and accumulate these values ​​into the unit heat value of each video segment's heat value statistical unit. This allows for objective statistical analysis of the unit heat values ​​of each heat value statistical unit in the first video, thereby enabling the determination of key video segments from the first video based on these unit heat values. The entire process, by first matching the video content of the second videos to determine video segments and then using the video heat values ​​of the second videos for heat analysis, achieves accurate determination of key video segments and improves the accuracy of key video segment location.

[0022] Sixthly, this application provides a video playback method, including:

[0023] Play the first video on the video playback interface;

[0024] In response to a time-prompt trigger event for a key video segment of the first video, the playback timeline of the first video is displayed on the video playback interface; the key video segment is determined based on the unit popularity value of each popularity value statistical unit in the first video; the unit popularity value of each popularity value statistical unit in the first video, the video popularity value of each of the multiple second videos related to the content of the first video, and the video segments in the first video that match each of the multiple second videos;

[0025] A key moment prompt is displayed at the start time of the segment corresponding to the key video segment on the playback timeline.

[0026] Seventhly, this application also provides a video playback device, comprising:

[0027] The video playback module is used to play the first video on the video playback interface;

[0028] A timeline display module is used to display the playback timeline of the first video on the video playback interface in response to a time prompt trigger event for a key video segment of the first video; the key video segment is determined based on the unit popularity value of each popularity value statistical unit in the first video; the unit popularity value of each popularity value statistical unit in the first video is obtained based on the video popularity values ​​of multiple second videos related to the content of the first video, and the video segments in the first video that match each of the multiple second videos;

[0029] The prompt display module is used to display a key moment prompt at the start time of the segment corresponding to the key video segment on the playback timeline.

[0030] Eighthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described video playback method.

[0031] Ninthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described video playback method.

[0032] In a tenth aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described video playback method.

[0033] The aforementioned video playback method, apparatus, computer equipment, computer-readable storage medium, and computer program product, in response to a time-prompt trigger event during the playback of a first video, display the playback timeline of the first video. At the start time of the segment corresponding to the key video segment on the playback timeline, a key moment prompt is displayed. When determining the key video segment, multiple second videos related to the content of the first video are acquired. From the first video, video segments matching each of the multiple second videos are determined, and the video popularity values ​​of each of the multiple second videos are acquired. These video popularity values ​​are accumulated into the unit popularity value of each video segment's popularity value statistical unit. This allows for objective statistical analysis of the unit popularity values ​​of each popularity value statistical unit in the first video, thereby enabling the determination of the key video segment from the first video based on these unit popularity values. The entire process, by first matching the video content of the second videos to determine video segments and then using the video popularity values ​​of the second videos for popularity analysis, enables accurate determination of the key video segment and improves the accuracy of key video segment location. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a diagram illustrating the application environment of a key video segment determination method in one embodiment.

[0036] Figure 2This is a flowchart illustrating a method for determining key video segments in one embodiment;

[0037] Figure 3 This is a schematic diagram illustrating the determination of a second video in one embodiment;

[0038] Figure 4 This is a schematic diagram illustrating the determination of similar video frame features in one embodiment;

[0039] Figure 5 This is a schematic diagram showing a border in the second video frame in one embodiment;

[0040] Figure 6 This is a schematic diagram showing a border in the second video frame in another embodiment;

[0041] Figure 7 This is a flowchart illustrating a method for determining key video segments in another embodiment;

[0042] Figure 8 Here is a film and television drama popularity-time graph in one embodiment;

[0043] Figure 9 This is a schematic diagram of highlight clip material in one embodiment;

[0044] Figure 10 This is a diagram illustrating the application environment of a video playback method in one embodiment;

[0045] Figure 11 This is a flowchart illustrating a video playback method in one embodiment;

[0046] Figure 12 This is a schematic diagram illustrating the switching of key video segments in one embodiment;

[0047] Figure 13 This is a structural block diagram of a key video segment determination device in one embodiment;

[0048] Figure 14 This is a structural block diagram of a video playback device in one embodiment;

[0049] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] In order to clearly describe the technical solution of this application and facilitate understanding of the technical solution of this application, the key concepts involved in this application will be explained below.

[0052] 1. First Video.

[0053] The first video refers to a video from which key video segments need to be identified. For example, the first video can specifically refer to movies, TV dramas, news, documentaries, live replays, sports event videos, etc., from which key video segments need to be identified.

[0054] 2. Second video.

[0055] The second video refers to a video whose content is related to the first video. For example, the second video could specifically be a short video related to the content of the first video, meaning a video shorter than the first video. Content relevance means that the second video shares a portion of the same content as the first video, either in terms of discussion or presentation.

[0056] 3. Video segment.

[0057] A video segment refers to a time segment in the first video that matches the content of the second video. Specifically, it can be the time range in the first video that matches the content of the second video, that is, the part between specific start and end times in the first video.

[0058] 4. Video popularity score.

[0059] Video popularity score is a comprehensive indicator that measures the popularity of a second video on a video platform. It can be calculated using various behavioral data of users on the video platform. For example, video popularity score can be calculated using at least one of the following: user viewing behavior data and interaction behavior data.

[0060] 5. Popularity value statistics unit.

[0061] A popularity score unit refers to the basic time unit used to measure the popularity of a video's content. For example, the popularity score unit can be a video frame or a second. A video frame is the smallest unit of video content, and each second typically contains multiple video frames. Calculating popularity scores based on video frames provides highly accurate analysis.

[0062] 6. Unit heat value.

[0063] Unit popularity value refers to the popularity index within a specific unit of popularity (such as video frame, second, etc.). It is a quantitative description of the popularity of video content within a fine-grained time range, reflecting the degree of attention users pay to the content within that time period.

[0064] 7. Key video clips.

[0065] Key video clips refer to segments in the first video that contain significant information and are likely to spark discussion and resonance among viewers; they can also be called highlight segments. For example, in the case of a film or television drama, key video clips might be scenes featuring humor, conflict, or plot twists. Similarly, in the case of a sports event, key video clips might be segments depicting winning a match or game.

[0066] The key video segment determination method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, the data storage system stores the data that server 102 needs to process. The data storage system can be set up independently, integrated into server 102, or placed in the cloud or on other network servers. Server 102 acquires multiple second videos related to the first video content, determines video segments matching each of the multiple second videos from the first video, identifies popularity value statistical units within each video segment of the first video, acquires the video popularity values ​​of each of the multiple second videos, accumulates the video popularity values ​​of the multiple second videos into the unit popularity value of each video segment's popularity value statistical unit, and determines key video segments from the first video based on the unit popularity values ​​of each popularity value statistical unit in the first video.

[0067] Among them, server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0068] In one exemplary embodiment, such as Figure 2 As shown, a method for determining key video segments is provided. This embodiment illustrates the application of this method to a server, but it is understood that the method can also be applied to a terminal, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes steps 202 to 210. Wherein:

[0069] Step 202: Obtain multiple second videos related to the content of the first video.

[0070] The first video refers to the video from which key video segments need to be identified. For example, the first video can specifically refer to movies, TV dramas, news reports, documentaries, live replays, sports event videos, etc., from which key video segments need to be identified.

[0071] The second video refers to a video related to the content of the first video. For example, a second video could specifically be a short video related to the content of the first video, meaning a video shorter than the first video. Content relevance means that the second video shares a portion of the content discussed or shown in the first video. For instance, if the first video is a film or television drama, the second video could specifically be a related video containing clips from that film or television drama. Similarly, if the first video is a news report, the second video could specifically be a related video containing news clips.

[0072] For example, when it is necessary to determine key video segments from a first video, the server retrieves multiple second videos related to the content of the first video. In specific applications, the server can utilize textual or image information describing the video characteristics of the first video to retrieve multiple second videos related to the content of the first video from multiple sample videos.

[0073] In practical applications, the server can utilize textual information describing the video features of the first video, such as the first video description information, to obtain multiple candidate videos related to the theme of the first video from multiple sample videos. Then, based on these candidate videos, multiple second videos related to the content of the first video can be obtained. Specifically, after obtaining multiple candidate videos, the server can directly use these candidate videos as multiple second videos related to the content of the first video, or it can use video frame matching to filter out multiple second videos related to the content of the first video from the multiple candidate videos. Video frame matching refers to extracting video frames from both the first video and the candidate videos and then performing matching.

[0074] Step 204: From the first video, determine the video segments that match each of the multiple second videos.

[0075] In this context, a video segment refers to a time interval within the first video that matches the content of the second video. Specifically, it can be the time range within the first video that matches the content of the second video, i.e., the portion between specific start and end times in the first video. For example, a video segment could specifically refer to the time interval from second X1 to second X2 in the first video, where X2 is greater than X1, and X1 is a positive number greater than or equal to 0. For instance, if the first video is a film or television series, the video segment could specifically refer to second X1 to second X2 in episode X, where X is a positive number greater than 0.

[0076] For example, the server will determine similar content from the first video that matches the content of each of the multiple second videos, and the video segments corresponding to the similar content that matches the content of each of the multiple second videos will be used as the video segments that match each of the multiple second videos.

[0077] In practical applications, similar content can specifically refer to similar video frames. For each second video, the server will determine the similar video frames from the first video that match the content of the second video, and then use the video segment corresponding to the similar video frames as the video segment matching the second video. Similar content can also refer to similar subtitles. In this case, for each second video, the server will determine the similar subtitles from the first video that match the content of the second video, and then use the video segment corresponding to the similar subtitles as the video segment matching the second video.

[0078] Step 206: Determine the heat value statistical unit in each video segment of the first video.

[0079] The popularity score unit refers to the basic time unit used to measure the popularity of the video content in the first video. For example, the popularity score unit can be a video frame or a second. A video frame is the smallest unit of video content, and each second typically contains multiple video frames. Calculating the popularity score in video frames provides very accurate analysis.

[0080] For example, the server will determine the heat value statistics unit in each video segment of the first video.

[0081] Step 208: Obtain the video popularity value of each of the multiple second videos, and accumulate the video popularity values ​​of each of the multiple second videos into the unit popularity value of the popularity value statistics unit of each video segment.

[0082] Video popularity score is a comprehensive indicator measuring the popularity of a second video on a video platform. It can be calculated using various user behavior data from the video platform. For example, video popularity score can be calculated using at least one of the following: user viewing behavior data and interaction behavior data. Viewing behavior data refers to various data generated by users while watching the second video. Specific examples include video views, video viewing duration, and video completion rate. Interaction behavior data refers to data generated by users interacting with the video content or the video platform while watching the second video. Specific examples include likes, comments, shares, and favorites.

[0083] Among them, the unit popularity value refers to the popularity index within a certain time period (such as video frame, second, etc.) within the popularity value statistical unit. It is a quantitative description of the popularity of video content within a fine-grained time range and reflects the degree of attention users pay to the content within that time period.

[0084] For example, the server will obtain the video popularity value of each of the multiple second videos, and accumulate the video popularity values ​​of the multiple second videos into the unit popularity value of the popularity value statistics unit of each video segment.

[0085] In practical applications, the video popularity value of the second video is calculated in advance. Specifically, the server acquires various behavioral data of users interacting with the second video on the video platform, and calculates the video popularity value based on this data. In a specific application, a popularity index can be pre-set when calculating the video popularity value, and then the calculation can be performed using this index. In this embodiment, the pre-set popularity index can be configured according to the actual application scenario.

[0086] For example, taking video views as a popularity metric, the server can directly use the video views of the second video as its popularity value. As another example, if the popularity metric includes video views, likes, comments, shares, and favorites, the server can directly use the sum of these metrics as the second video's popularity value.

[0087] In a specific application, taking a TV series as an example, if the second video has a popularity value of 5000, and the video segment is from the 1000th to the 1100th second of episode 2, with the popularity value calculated in seconds, then the server will accumulate the video popularity value of 5000 into the unit popularity value for each second within that segment of episode 2. To further illustrate, assuming the unit popularity value for the 1000th second of episode 2 was 1000 before accumulation, then after accumulation, the unit popularity value for the 1000th second of episode 2 will become 6000. Assuming the unit popularity value for the 1100th second of episode 2 was 3000 before accumulation, then after accumulation, the unit popularity value for the 1100th second of episode 2 will become 8000.

[0088] Step 210: Determine key video segments from the first video based on the unit popularity values ​​of each popularity value statistical unit in the first video.

[0089] Key video clips refer to segments in the first video that contain significant information and are likely to spark discussion and resonance among viewers; they can also be called highlight segments. For example, if the first video is a film or television drama, key video clips could be segments featuring comedic moments, conflicts, or plot twists. Similarly, if the first video is a sports event video, key video clips could be segments depicting events such as winning a match or a game.

[0090] For example, the server will filter multiple target popularity statistical units from the popularity statistical units in the first video based on the unit popularity value of each popularity statistical unit in the first video, and then use the multiple target popularity statistical units to determine the key video segments from the first video.

[0091] In practical applications, the server can compare a preset popularity threshold with the unit popularity values ​​of each popularity statistical unit in the first video. From the popularity statistical units in the first video, it can filter out the popularity statistical units whose unit popularity values ​​are greater than the preset popularity threshold, and use them as target popularity statistical units. The preset popularity threshold can be configured according to the actual application scenario; for example, the preset popularity threshold can be 5000, 8000, etc.

[0092] In practical applications, the server can sort the unit popularity values ​​of each popularity statistical unit in the first video. Based on the sorting results and a preset selection ratio, it selects the group of popularity statistical units with the highest unit popularity values ​​from all popularity statistical units in the first video as the target popularity statistical units. The preset selection ratio can be configured according to the actual application scenario. For example, the preset selection ratio can be 10%, 15%, etc.

[0093] The aforementioned method for determining key video segments involves acquiring multiple second videos related to the content of a first video, identifying video segments from the first video that match each of the second videos, determining the popularity statistics units within each video segment of the first video, obtaining the video popularity values ​​of each of the second videos, and accumulating these values ​​into the unit popularity value of each video segment's popularity statistics unit. This allows for objective statistical analysis of the unit popularity values ​​of each popularity statistics unit in the first video, thereby enabling the determination of key video segments from the first video based on these unit popularity values. The entire process, by first matching the video content of the second videos to determine video segments and then using the video popularity values ​​of the second videos for popularity analysis, achieves accurate determination of key video segments and improves the accuracy of key video segment location.

[0094] In one exemplary embodiment, acquiring a plurality of second videos related to the first video content includes:

[0095] Retrieve multiple candidate videos related to the first video topic;

[0096] Extract multiple first video frames from the first video, and extract multiple second video frames from each candidate video respectively;

[0097] Each candidate video's multiple second video frames are matched with multiple first video frames to obtain the video frame matching result for each candidate video.

[0098] Based on the video frame matching results of each candidate video, multiple second videos related to the content of the first video are obtained.

[0099] Candidate videos refer to videos related to the theme of the first video. For example, a candidate video could specifically be a short video related to the theme of the first video, meaning a video shorter than the first video. Theme relevance means that the candidate videos discuss or showcase the same or similar themes as the first video. For instance, if the first video is a film or television series, then candidate videos could specifically be videos discussing that film or television series. Similarly, if the first video is a news report, then candidate videos could specifically be videos discussing that news report.

[0100] For example, the server will obtain multiple candidate videos related to the first video topic, extract multiple first video frames from the first video, and extract multiple second video frames from each candidate video. The multiple second video frames of each candidate video will be matched with the multiple first video frames to obtain the video frame matching result of each candidate video. Then, based on the video frame matching result of each candidate video, multiple candidate videos related to the content of the first video will be determined from the multiple candidate videos.

[0101] In specific applications, the frame extraction method when extracting video frames from the first video and candidate videos can be configured according to the actual application scenario. For example, the frame extraction method can be to sample at 5 frames per second, that is, for each second of video, the server will extract 5 video frames.

[0102] In practical applications, for each candidate video, by performing video frame matching between multiple second video frames of the candidate video and multiple first video frames respectively, it can be determined whether there are similar video frames among the multiple first video frames that are similar to any second video frame, thus obtaining the video frame matching result of the candidate video.

[0103] In a specific application, if the video frame matching result of the candidate video shows that there are similar video frames and the number of similar video frames is at least two, the server will determine the order of occurrence of the first video frames among the at least two similar video frames in the first video, and determine the order of occurrence of the second video frames among the candidate video that are similar to each of the at least two similar video frames. If the order of occurrence of the first video frames is the same as the order of occurrence of the second video frames, the server will treat the candidate video as the second video related to the content of the first video.

[0104] For example, such as Figure 3As shown, assuming the candidate video's frame matching result shows the existence of three similar video frames, and in the first video, the order of the first video frames among the three similar video frames is similar video frame 1 - similar video frame 2 - similar video frame 3, and similar video frame 1 is similar to second video frame 1, similar video frame 2 is similar to second video frame 2, and similar video frame 3 is similar to second video frame 3, if in the candidate video, the order of the second video frames similar to the three similar video frames is second video frame 1 - second video frame 2 - second video frame 3, then the candidate video can be considered a second video related to the content of the first video. If in the candidate video, the order of the second video frames similar to the three similar video frames is second video frame 3 - second video frame 1 - second video frame 2, then the candidate video can be considered not a second video related to the content of the first video. Figure 3 In the diagram, the dashed double arrow between the similar video frame and the first video indicates the video time at which the similar video frame appears in the first video, and the dashed double arrow between the second video frame and the candidate video indicates the video time at which the second video frame appears in the candidate video.

[0105] In a specific application, if the video frame matching result of the candidate video shows that there are similar video frames and the number of similar video frames is only one, or if the video frame matching result of the candidate video shows that there are no similar video frames, the server will consider the candidate video to be a second video that is not related to the content of the first video.

[0106] In this embodiment, based on obtaining multiple candidate videos related to the first video theme, extracting multiple first video frames from the first video, and extracting multiple second video frames from each candidate video, the video frame matching result of each candidate video can be obtained by matching the multiple second video frames of each candidate video with the multiple first video frames. Then, based on the video frame matching result of each candidate video, multiple second videos related to the content of the first video can be determined.

[0107] In an exemplary embodiment, multiple second video frames of each candidate video are respectively matched with multiple first video frames to obtain a video frame matching result for each candidate video, including:

[0108] Feature extraction is performed on multiple first video frames to obtain features of multiple first video frames;

[0109] For each candidate video, feature extraction is performed on multiple second video frames of the candidate video to obtain multiple second video frame features;

[0110] The video frame matching results of the candidate video are obtained by matching the features of multiple second video frames with the features of multiple first video frames.

[0111] The first video frame feature refers to the feature extracted from the first video frame. The second video frame feature refers to the feature extracted from the second video frame.

[0112] For example, when performing video frame matching, the server extracts features from multiple first video frames to obtain first video frame features corresponding to each first video frame, i.e., multiple first video frame features. For each candidate video, the server extracts features from multiple second video frames to obtain second video frame features corresponding to each second video frame, i.e., multiple second video frame features. Then, the server performs feature matching between the multiple second video frame features of the candidate video and the multiple first video frame features to obtain feature matching results for each of the multiple second video frame features. Finally, based on the feature matching results for each of the multiple second video frame features, the server obtains the video frame matching result of the candidate video.

[0113] In practical applications, for each second video frame feature of a candidate video, the server performs feature matching between the second video frame feature and multiple first video frame features. This determines the frame feature similarity between the second video frame feature and each of the multiple first video frame features. Then, using a similarity threshold and the frame feature similarity between the second video frame feature and each of the multiple first video frame features, it can be determined whether there are similar video frame features among the multiple first video frame features that are similar to the second video frame feature, thus obtaining the feature matching result for the second video frame feature. The similarity threshold can be configured according to the actual application scenario; for example, the similarity threshold could be 90%, 85%, etc.

[0114] In a specific application, taking five first video frames as an example, such as... Figure 4 As shown, during video frame matching, the server extracts features from each of the five first video frames, obtaining the first video frame features corresponding to each of the five first video frames. For each second video frame feature of the candidate video, the server performs feature matching between the second video frame feature and the five first video frame features to determine the frame feature similarity between the second video frame feature and each of the five first video frame features (e.g., ...). Figure 4 The values ​​shown are 50%, 55%, 60%, 80%, and 60%, respectively. When any frame feature similarity is greater than or equal to a similarity threshold (assumed to be 80%), the first video frame feature with a similarity greater than the threshold is considered a similar video frame feature to the second video frame feature (e.g., ...). Figure 4The fourth feature of the first video frame is shown. The feature matching result of the second video frame feature is that there are similar video frame features.

[0115] In practical applications, based on the feature matching results of multiple second video frame features, the server can determine whether each second video frame in the candidate video has a similar video frame, thus obtaining the video frame matching result of the candidate video. If the feature matching result of each second video frame feature is that no similar video frame feature exists, then the video frame matching result of the candidate video can be determined to be that no similar video frame exists. If the feature matching result of any second video frame feature is that a similar video frame feature exists, the server will take the first video frame corresponding to the similar video frame feature as the similar video frame corresponding to the second video frame of that second video frame feature, and obtain the video frame matching result of the candidate video based on the similar video frames of at least one second video frame.

[0116] In practical applications, both the first video and the candidate videos consist of multiple frames of images. Each frame can be represented as a vector, i.e., a video frame feature. There are various methods for calculating video frame features. In this embodiment, the method for calculating video frame features is not limited and can be selected according to the actual application scenario. For example, specific methods for calculating video frame features may include, but are not limited to, feature extraction methods such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), and HOG (Histogram of Oriented Gradients), as well as feature extraction models based on deep neural networks and image feature extraction methods based on Transformer (a deep learning model based on an attention mechanism).

[0117] In a specific application, this embodiment utilizes a pre-trained CLIP (Contrastive Language–Image Pre-training) model to extract features from video frames, obtaining video frame features. The CLIP model is a multimodal model based on parallel image and text processing, and then constructs the training objective by calculating the similarity between the feature vectors of the two branches. In this embodiment, the image encoder in the CLIP model can be used to extract features from video frames, obtaining video frame features.

[0118] In this embodiment, accurate video frame matching of the first video frame and the second video frame can be achieved by performing feature matching on the features of the first video frame and the features of the second video frame, thereby obtaining the video frame matching result of the candidate video.

[0119] In an exemplary embodiment, feature extraction is performed on multiple second video frames of the candidate video to obtain multiple second video frame features, including:

[0120] For each second video frame of the candidate video, perform border detection on the second video frame to obtain the border detection results;

[0121] Based on the border detection results, the region where the video frame is located in the second video frame is determined, and feature extraction is performed on the region where the video frame is located to obtain the features of the second video frame.

[0122] For example, since the candidate videos are obtained from the video platform, some videos may have borders added. In order to remove the influence of these borders, for each second video frame of the candidate video, the server needs to first perform border detection on the second video frame, obtain the border detection result, and then determine the area where the video screen is located in the second video frame based on the border detection result, so as to extract the effective screen area in the second video frame. Then, feature extraction is performed on the area where the video screen is located to obtain the features of the second video frame.

[0123] In practical applications, the server can use a pre-trained object detection model to perform border detection on the second video frame, obtaining border detection results. These results include the location of the border and its confidence score. If the border confidence score indicates the presence of a border in the second video frame, the server will crop the frame based on the border's location to obtain the video frame's area. If the border confidence score indicates the absence of a border, the server can directly use the entire second video frame as the video frame's area.

[0124] In a specific application, the object detection model used can be configured according to the actual application scenario. For example, the object detection model could be YOLOv8 (You Only Look Once-v8, an object detection model). When determining whether a border exists in the second video frame based on the border confidence score, the server compares the border confidence score with a confidence threshold. If the border confidence score is greater than or equal to the confidence threshold, it is determined that a border exists in the second video frame. The confidence threshold can be configured according to the actual application scenario. For example, if the confidence threshold is 0.75, then if the border confidence score is greater than 0.75, it can be considered that a border exists in the second video frame.

[0125] In a specific application, the position of the border refers to its location within the second video frame. This can be defined by four parameters: the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the width of the border, and the height of the border. The x-coordinate and y-coordinate are determined by the pixel coordinates of the top-left corner pixel, while the width and height are in pixels. For example, assuming the border's position is: x-coordinate = 100, y-coordinate = 50, width = 200, height = 150, then the top-left corner of the border is located at pixel (100, 50), with a width of 200 pixels and a height of 150 pixels.

[0126] In a specific application, an example is given to illustrate the case where a border exists in the second video frame. For example... Figure 5 As shown, the second video frame may include the area where the video frame is located and a text description of the video frame. Specifically, as... Figure 5 As shown, for the second video frame (a), it includes the area where the video frame is located and the text description surrounding the area of ​​the video frame (such as...). Figure 5 The diagram includes text descriptions 1 and 2. By detecting the border, the location 502 of the border and the border confidence level (e.g., ...) can be determined. Figure 5 (As shown in 0.99). For the second video frame (b), it also includes the area where the video frame is located and the text description surrounding that area (e.g., ...). Figure 5 The diagram includes text descriptions 3 and 4. By detecting the border, the location of the border (504) and the border confidence level (e.g., ...) can be determined. Figure 5 (As shown, 0.80).

[0127] In a specific application, an example is given to illustrate the case where a border exists in the second video frame. For example... Figure 6 As shown, the second video frame may include both the area where the video frame is located and the background area. Specifically, as... Figure 6 As shown, for the second video frame (c), which includes the area where the video frame is located and the video background areas on the left and right sides of the video frame area, the location of the border 602 and the border confidence (e.g., ...) can be determined through border detection. Figure 6 The value shown is 0.88. For the second video frame (d), which also includes the area where the video frame is located and the video background areas on the left and right sides of that area, border detection can determine the border location (604) and the border confidence score (e.g., ...). Figure 6 (As shown, 0.76). It should be noted that, as... Figure 6 As shown, apart from the border with a confidence level of 0.76, for items such as... Figure 6The second video frame (d) shown can also detect borders with a confidence level of 0.58 and 0.34. These two borders are discarded because their confidence levels are lower than the border with a confidence level of 0.76. For the second video frame (e), which also includes the video frame area and the video background areas on the left and right sides of the video frame area, border detection can determine the location of the border (606) and its confidence level (e.g., ...). Figure 6 The value shown is 0.86). For the second video frame (f), which includes the area containing the video frame and the video background areas above and below it, border detection can determine the border location (608) and the border confidence level (e.g., ...). Figure 6 (As shown, it is 0.88). It should be noted that... Figure 6 The background area of ​​the medium video is shown in black.

[0128] In this embodiment, for each second video frame of the candidate video, by performing border detection on the second video frame, the border detection result can be obtained. Then, based on the border detection result, the area where the video screen is located in the second video frame can be determined, and the feature of the area where the video screen is located can be extracted to obtain the accurate features of the second video frame.

[0129] In one exemplary embodiment, multiple candidate videos related to a first video topic are obtained, including:

[0130] Obtain the first video description information of the first video;

[0131] Based on the description information of the first video, multiple candidate videos related to the topic of the first video are retrieved.

[0132] The first video description information refers to the text information used to summarize and describe the first video. For example, the first video description information can specifically be at least one of the following: video title or video synopsis. For instance, if the first video is a film or television series, the first video description information could specifically be the film / television series title and a general synopsis. The general synopsis provides an overall overview of the film / television series.

[0133] For example, the server obtains the first video description information of the first video and retrieves multiple candidate videos related to the theme of the first video based on the first video description information. In specific applications, the server performs correlation matching between the first video description information and the sample video description information of multiple sample videos to retrieve multiple candidate videos related to the theme of the first video.

[0134] In this embodiment, by obtaining the first video description information of the first video, video retrieval can be performed using the first video description information to obtain multiple candidate videos related to the theme of the first video.

[0135] In an exemplary embodiment, retrieving multiple candidate videos related to a first video topic based on first video description information includes:

[0136] Obtain the sample video description information for each of the multiple sample videos;

[0137] The first video description information is matched with the sample video description information of each of the multiple sample videos to obtain the correlation matching results.

[0138] Based on the correlation matching results, multiple candidate videos related to the theme of the first video are obtained from multiple sample videos.

[0139] The sample video description information refers to the text information used to describe the sample video. For example, the sample video description information may include the sample video title, a brief description, etc.

[0140] For example, when performing video retrieval, the server retrieves sample video description information for each of multiple sample videos from the video library of the video platform. The first video description information is then matched with the sample video description information for each of the multiple sample videos to obtain a correlation matching result. Based on the correlation relationship between each sample video description information and the first video description information in the correlation matching result, the sample video description information associated with the first video description information is determined. From the multiple sample videos, the sample videos whose sample video description information is associated with the first video description information are selected as candidate videos related to the theme of the first video, thus obtaining multiple candidate videos.

[0141] In practical applications, the server can utilize a pre-trained large language model (LLM) to calculate the correlation between sample video description information and primary video description information, i.e., to determine whether the sample video description information and primary video description information are related. By inputting the correlation between the sample video description information and primary video description information into the large language model, along with appropriate prompt words, the large language model can output the correlation between the sample video description information and primary video description information, which can be either related or unrelated. The large language model (LLM) is a language model composed of a Transformer network with a large number of parameters (typically billions or more weights), trained on a large amount of unlabeled text using self-supervised or semi-supervised learning.

[0142] In this embodiment, by obtaining the sample video description information of each of the multiple sample videos, and performing correlation matching between the first video description information and the sample video description information of each of the multiple sample videos, a correlation matching result can be obtained. Based on the correlation matching result, multiple candidate videos related to the theme of the first video can be selected from the multiple sample videos.

[0143] In one exemplary embodiment, determining video segments from the first video that match each of a plurality of second videos includes:

[0144] From the first video, identify at least two similar video frames that match the content of each of the multiple second videos, and determine the video moments of each of the at least two similar video frames;

[0145] For each second video, a video segment matching the content of the second video is determined based on the video times of at least two similar video frames that match the content of the second video.

[0146] For example, when determining a video segment, the server will identify at least two similar video frames from the first video that match the content of each of the multiple second videos, and determine the video time of each of the at least two similar video frames. For each second video, based on the video time of each of the at least two similar video frames that match the content of the second video, the server will determine a video segment that matches the content of the second video.

[0147] In specific applications, for each second video, after determining the video timestamps of at least two similar video frames, the server determines the video timestamps of the first and last similar video frames, and identifies the time segment between the video timestamps of the first and last similar video frames as the video segment that matches the content of the second video.

[0148] In a specific application, taking a TV series as an example, if the video time of the first similar video frame is the first frame of the 1000th second of episode 2, and the video time of the last similar video frame is the first frame of the 1100th second of episode 2, then the server will determine the video segment from the 1000th second to the 1100th second of episode 2 in the first video as the video segment that matches the content of the second video. In this video segment, the 1000th second of episode 2 is the start time, and the 1100th second of episode 2 is the end time.

[0149] In this embodiment, by determining at least two similar video frames from the first video that match the content of each of the multiple second videos, and determining the video time of each of the at least two similar video frames, the determination of the video segment matching the content of the second video can be achieved by utilizing the video time of each of the at least two similar video frames.

[0150] In an exemplary embodiment, determining key video segments from the first video based on the unit popularity values ​​of each popularity value statistical unit in the first video includes:

[0151] Based on the unit popularity value of each popularity value statistical unit in the first video, multiple target popularity value statistical units are selected from each popularity value statistical unit in the first video;

[0152] From multiple target heat value statistical units, determine at least one heat value statistical unit segment;

[0153] Based on at least one heat value statistical unit segment, key video segments are determined from the first video.

[0154] Among them, the heat value statistical unit segment refers to a segment composed of a portion of target heat value statistical units that are continuous in time.

[0155] For example, the server can filter multiple target popularity value statistical units from each popularity value statistical unit in the first video by comparing a preset popularity value threshold with the unit popularity value of each popularity value statistical unit in the first video, or by sorting the unit popularity values ​​of each popularity value statistical unit in the first video, and then determine at least one popularity value statistical unit segment that is continuous in time from the multiple target popularity value statistical units, so as to use at least one popularity value statistical unit segment to determine key video segments from the first video.

[0156] In practical applications, when multiple target popularity value statistical units are determined, the server will use the timestamps of each target popularity value statistical unit to determine whether each target popularity value statistical unit is continuous in time with its adjacent target popularity value statistical units, so as to determine at least one popular value statistical unit segment that is continuous in time from multiple target popularity value statistical units.

[0157] In a specific application, for each target popularity value statistical unit, the server compares the timestamp of the target popularity value statistical unit with the timestamp of the adjacent target popularity value statistical units. If the timestamp difference is less than or equal to a preset timestamp difference threshold, the target popularity value statistical unit is considered to be continuous in time with the adjacent target popularity value statistical units.

[0158] The preset timestamp difference threshold can be configured according to the actual application scenario, in conjunction with the popularity value statistics unit. For example, if the popularity value statistics unit is a video frame, the preset timestamp difference threshold can be set based on the number of video frames extracted per second. Specifically, the preset timestamp difference threshold can be the extraction interval of video frames per second. Further, if the number of video frames extracted per second is 5, the timestamp difference threshold can be 200 milliseconds. Or, if the popularity value statistics unit is a second, the preset timestamp difference threshold can be set to 1 second.

[0159] In practical applications, after identifying at least one popularity value statistical unit segment, the server determines the number of popularity value statistical unit segments. If there is only one popularity value statistical unit segment, the server extracts the corresponding video segment from the first video as the key video segment. If there are multiple popularity value statistical unit segments, the server needs to extract the corresponding video segments from each of the multiple popularity value statistical unit segments from the first video, and then obtain the key video segment based on these multiple video segments.

[0160] In this embodiment, by utilizing the unit popularity value of each popularity value statistical unit in the first video, multiple target popularity value statistical units can be filtered out from each popularity value statistical unit in the first video. Thus, multiple target popularity value statistical units can be used to determine at least one popularity value statistical unit segment, and then at least one popularity value statistical unit segment can be used to determine key video segments from the first video.

[0161] In one exemplary embodiment, determining key video segments from a first video based on at least one heat value statistical unit segment includes:

[0162] When there are multiple popularity value statistical unit segments, for each popularity value statistical unit segment, the segment video segment corresponding to the popularity value statistical unit segment is extracted from the first video;

[0163] Multiple video segments are integrated to obtain multiple candidate video segments;

[0164] Based on the duration of each of the multiple candidate video segments, key video segments are selected from the multiple candidate video segments.

[0165] For example, when there are multiple popularity value statistical unit segments, for each popularity value statistical unit segment, the server will extract the segment video segment corresponding to the popularity value statistical unit segment from the first video, and then integrate the segment content of multiple segment video segments to obtain multiple candidate video segments. Then, using the segment duration of each of the multiple candidate video segments, the key video segment is selected from the multiple candidate video segments.

[0166] In practical applications, if the popularity value statistical unit is a video frame, then the popularity value statistical unit segment can be in the form of frame m at second M to frame n at second N. The server can extract the video content from frame m at second M to frame n at second N from the first video as the segment video corresponding to the popularity value statistical unit segment. If the popularity value statistical unit is a second, then the popularity value statistical unit segment can be in the form of second M to second N. The server can extract the video content from second M to second N from the first video as the segment video corresponding to the popularity value statistical unit segment. Here, M is less than N, and M, m, N, and n are all positive numbers greater than 0.

[0167] In practical applications, when integrating the content of multiple video segments, the server calculates the time interval between each video segment and its adjacent video segments. Then, using the time interval, it determines at least two video segments that need to be merged. The server merges these two video segments to obtain a merged video segment. Finally, each merged video segment and the unmerged video segments are used as multiple candidate video segments.

[0168] In practical applications, after obtaining multiple candidate video clips, the server compares the duration of each clip with a preset duration threshold to filter out clips shorter than the threshold, thus obtaining the key video clip. The preset duration threshold can be configured according to the actual application scenario. For example, if the preset threshold is 3 seconds, the server will filter out clips shorter than 3 seconds and retain the remaining clips as the key video clip.

[0169] In this embodiment, when there are multiple heat value statistical unit segments, for each heat value statistical unit segment, the segment video segment corresponding to the heat value statistical unit segment can be extracted from the first video. Then, by integrating the segment content of multiple segment video segments, multiple candidate video segments can be obtained. Based on the segment duration of each of the multiple candidate video segments, key video segments can be selected from the multiple candidate video segments to achieve the extraction of key video segments.

[0170] In an exemplary embodiment, multiple video segments are integrated to obtain multiple candidate video segments, including:

[0171] For each video segment, determine the time interval between the targeted video segment and adjacent video segments;

[0172] If the time interval between segments is less than a preset time interval threshold, the targeted video segment and the adjacent video segments will be merged to obtain a merged video segment.

[0173] Each merged video segment and each unmerged segment of video video are treated as multiple candidate video segments.

[0174] For example, when merging video segments, for each segment, the server determines the time interval between the targeted video segment and adjacent video segments (i.e., the time interval between the two video segments appearing in the first video). It then compares this time interval with a preset time interval threshold. If the time interval is less than the preset threshold, it indicates that the targeted video segment and adjacent video segments can be merged. The server then merges the targeted video segment and adjacent video segments to obtain a merged video segment. After determining whether all video segments need to be merged, each merged video segment and the unmerged video segments are considered as multiple candidate video segments. The preset time interval threshold can be configured according to the actual application scenario. For example, a preset time interval threshold of 3 seconds means that if the time interval is less than 3 seconds, the two video segments can be merged.

[0175] In practical applications, when merging the targeted video segment with adjacent video segments, the server will supplement the missing video content between the targeted video segment and the adjacent video segments based on the first video. That is, the missing video content between the targeted video segment and the adjacent video segments will be merged together with the targeted video segment and the adjacent video segments to obtain the merged video segment.

[0176] In a specific application, taking a TV series as the first video and a preset time interval threshold of 3 seconds as an example, assuming the targeted video segment is from 300 seconds to 320 seconds of episode 2, and the adjacent video segment is from 322 seconds to 350 seconds of episode 2, the calculated segment time interval is 2 seconds, which is less than the preset time interval threshold. The server will merge the targeted video segment with the adjacent video segment and supplement the content from 320 seconds to 322 seconds of episode 2 during the merging process. That is, the merged video segment includes the video content from 300 seconds to 350 seconds of episode 2.

[0177] In this embodiment, for each segment of video, by determining the segment time interval between the targeted segment of video and adjacent segment of video, it is possible to use the segment time interval to determine whether the segment of video needs to be merged. Thus, if the segment time interval is less than a preset time interval threshold, the targeted segment of video and adjacent segment of video can be merged to obtain a merged video segment. Furthermore, each merged video segment and unmerged segment of video can be used to determine multiple candidate video segments.

[0178] In one exemplary embodiment, the key video segment determination method further includes:

[0179] Obtain the description information of the second video from the first video;

[0180] The second video description information and key video segments are input into a pre-trained video segment description model, which then predicts the segment description information of the key video segments.

[0181] The second video description information refers to the textual information used to describe the first video in detail, based on key video clips, and is usually different from the first video description information used to summarize the first video. For example, the second video description information may specifically include a video synopsis related to the key video clips, information about the characters in the video, etc. Here, the video synopsis related to the key video clips can include an overall video synopsis of the first video and a partial video synopsis of the key video clip. For example, if the first video is a film or television series, the second video description information may specifically include an overall plot synopsis and a partial plot synopsis related to the key video clips, as well as information about the characters in the film or television series. Further, a partial plot synopsis may be a synopsis of the episode containing the key video clip. Information about the characters in the film or television series may include information about the characters and the corresponding actors. Clip description information refers to the textual information used to describe key video clips. For example, clip description information may specifically include a clip title used to describe the key video clip.

[0182] For example, after extracting the key video segments, the server obtains the second video description information of the first video, inputs the second video description information and the key video segments into a pre-trained video segment description model, and predicts the segment description information of the key video segments through the pre-trained video segment description model.

[0183] The pre-trained video segment description model can be configured according to the actual application scenario. For example, the pre-trained video segment description model can be a pre-trained multimodal large model based on a large language model. When the second video description information and key video segments are input into the pre-trained video segment description model, and appropriate prompt words are also input, the video segment description model can output segment description information.

[0184] In this embodiment, based on the second video description information of the first video, by inputting the second video description information and key video segments into a pre-trained video segment description model, the pre-trained video segment description model can be used to accurately predict the segment description information of key video segments.

[0185] In an exemplary embodiment, taking the application of the key video segment determination method to extract highlight segments from films and television dramas as an example, this application also provides an application scenario in which the above-mentioned key video segment determination method is applied. Specifically, the application of the key video segment determination method in this application scenario is as follows:

[0186] Highlight scenes in film and television dramas refer to segments that are rich in information and resonate with audiences, often representing crucial moments in the plot. Scenes featuring humor, conflict, or plot twists are more likely to become highlight scenes. By identifying these highlight scenes, downstream content distribution channels, such as recommendation systems and content operations, can be provided with video content features at different granularities, improving distribution efficiency while significantly reducing the cost of manual highlight scene discovery.

[0187] Traditional highlight fragment mining methods mainly fall into two categories: supervised models based on manually labeled data and unsupervised models. The supervised model approach involves using multimodal techniques to calculate features for each video frame. This supervised model, essentially a multi-classification model, estimates the final constructed image and selects the frame with the highest probability as the starting video frame. Then, starting from this starting frame, a fixed length of 128 video frames is selected as highlight fragments. The specific approach to mining using an unsupervised model can be an unsupervised learning method that uses video duration as an implicit supervisory signal. This method assumes that shorter videos have higher information content, so the segments cut out can be considered highlight segments. Conversely, longer videos have lower information content, and the segments cut out are not highlight segments. This approach divides the training samples D into three parts, namely D={DS, DL, DR}, where DS represents the set of short videos, DL represents the set of long videos, and videos shorter than 15s are defined as short videos, while videos longer than 45s are defined as long videos. A pair-wise method (operating on or comparing each pair of elements in a set of data) is used to construct samples, that is, one video segment is taken from each of the video segments cut out from DS and DL to form a pair of samples (si, sj). Then, the difference between the two parts is calculated based on the ranking loss, and the confidence of the sample is identified by a trainable binary latent variable.

[0188] However, traditional methods using supervised models based on manually labeled data suffer from drawbacks. Manual labeling leads to high costs and poor maintainability. The definition of highlight segments varies from person to person, resulting in significant subjectivity during labeling. Furthermore, the cost and difficulty of labeling limit the frequency of data updates, greatly impacting industry applications. Additionally, videos from different fields vary greatly; for example, variety shows and dramas, live streams and movies differ significantly in visuals and plots, requiring separate data labeling for each field, which is difficult to cover all. Unsupervised models, which assume that only short videos contain highlights and long videos don't, introduce noise, creating a new noise reduction problem. Moreover, model performance is heavily dependent on the training dataset, failing to truly address the practical needs of industry applications.

[0189] Based on this, this application proposes a novel solution: using topic-related short videos (i.e., candidate videos) to match segments in the original video (i.e., the first video), and using the accumulated posterior consumption signal (i.e., video popularity value) from content-related short videos (i.e., the second video) to locate highlight segments. This method requires no manual annotation or training dataset, significantly improving the efficiency and accuracy of highlight segment mining. Under parallel computing conditions, it can greatly reduce the latency of highlight segment mining. Furthermore, the longer the TV series / movies are played and the more consumption data is generated, the more accurate the extracted highlight segments become. Based on this, training data can also be accumulated for training other models.

[0190] Specifically, the key video segment determination method of this application mainly includes steps such as relevant short video mining, video feature vector calculation, mapping of relevant short videos to the original film and television drama, calculation of the popularity curve of the film and television drama, and highlight segment localization. The specific technologies involved in each step are as follows: First, the relevant short video mining part adopts correlation calculation based on a large model to find short videos (i.e., candidate videos) related to the film and television drama to be processed (i.e., the first video) across the entire network. Then, feature vectors are calculated for each frame of the relevant short videos and their corresponding film and television drama videos (specifically, the CLIP model can be used to calculate video frame features), to obtain video frame features. After obtaining the video frame features, frame-by-frame matching is performed on the relevant short videos and the original film and television drama to obtain the mapping relationship between the content-related short videos (second videos) and the long videos of the film and television drama (first videos). Each long video corresponds to multiple content-related short videos, so the popularity of the content-related short videos is accumulated and added to the long video segments of the film and television drama that match them to obtain the popularity of the long video at each moment. The popularity is divided according to a certain threshold to obtain some high-popularity video segments, and these segments are then post-processed to obtain highlight segments. The generated highlight clips can be published as complete short videos, or they can display heat or highlight indicators at the corresponding progress points in the original video, improving the user experience.

[0191] It should be noted that the data used in this application can be sourced from video platforms, primarily including popular TV dramas and related short videos, as well as data such as the number of views for these short videos. For each popular TV drama, data such as the number of related content items and views / plays can be recorded. In this application, the main data used are the number of content items and views / plays. The number of content items includes the number of related short videos, and views / plays include the number of views for these short videos.

[0192] In one exemplary embodiment, such as Figure 7As shown, the key video segment determination method of this application mainly includes related short video (i.e., candidate video) mining, video feature calculation (including series frame feature calculation and short video frame feature calculation), short video to original film and television drama (i.e., first video) matching and heat mapping, highlight segment cropping, and highlight segment title generation based on a large model. The technical details of each part are described in detail below.

[0193] In specific applications, in the relevant short video mining part, this application will obtain the sample video description information of each of the multiple sample short videos, and perform correlation matching between the first video description information and the sample video description information of each of the multiple sample short videos to obtain the correlation matching result. Based on the correlation matching result, multiple candidate videos related to the theme of the first video will be obtained from the multiple sample short videos.

[0194] In a specific application, a pre-trained large language model can be used to calculate the correlation between sample video description information and first video description information. By inputting the correlation between sample video description information and first video description information into the large language model, and simultaneously inputting appropriate prompt words, the large language model can output the correlation between sample video description information and first video description information, which can be either related or unrelated.

[0195] For example, the first video description information could be the title and synopsis of a TV series, while the sample video description information could be the title of a short video. By inputting the TV series title, synopsis, and short video title into a pre-trained large language model and adjusting appropriate prompts, the model can be guided to determine whether a short video is related to a TV series. To further illustrate, if a TV series is named ABCC and contains characters A, B, and C, then the title of a relevant short video could be a title containing the TV series title, or a title containing at least one of characters A, B, or C.

[0196] In practical applications, the video feature calculation mainly involves calculating the features of episode frames and short video frames. To match short videos (candidate videos) with the original content of TV dramas (first videos), and to find the corresponding video moment of the original video (first video) for each short video, it is necessary to calculate the video frame features (i.e., first video features) of each video frame (first video frame) in each episode of the TV drama and the video frame features (i.e., second video frames) of each short video. This application uses the CLIP model to calculate video frame features. First, sampling is performed at 5 frames per second (i.e., extracting multiple first video frames), and then the video frame features of each sampled frame are calculated. After the first video frame features of the original video are calculated, they are saved in the format of episode number + time (seconds) + frame number, and an index is built using Faiss (a vector retrieval library), i.e., an index of the feature vectors of each episode frame of the long video is created. For short videos, which often involve significant manual editing, borders may be added. To remove the influence of these borders, it's necessary to extract the effective frame area (i.e., the area containing the video frame) and calculate frame features (i.e., the second video frame features) for this area. This application uses YOLOv8 for border detection and performs effective frame cropping, then calculates frame features on the cropped frame. When calculating short video frame features, sampling is performed at 5 frames per second, and CLIP is used to calculate frame features. The frame features are then saved according to the short video identifier (specifically, the short video ID). The YOLOv8 border detection model is trained using a manually labeled dataset. This application performs large-scale sampling on accumulated short video data to cover as many types of borders as possible.

[0197] In practical applications, the matching and popularity mapping of short videos to the original TV series (i.e., the first video) first initializes the initial popularity sequence of each episode of the TV series to 0. For each short video related to the TV series (i.e., each candidate video), the Faiss vector retrieval is used to query the corresponding time in the long video of the TV series. That is, the short video frame features are matched with the TV series episode vectors. In other words, for each candidate video, the multiple second video frame features of the candidate video are matched with the multiple first video frame features respectively to obtain the video frame matching results of the candidate video. Based on the video frame matching results of each candidate video, the second video related to the content of the first video is obtained. From the first video, at least two similar video frames that match the content of each of the multiple second videos are determined, and the video time of each of the at least two similar video frames is determined. For each second video, based on the video time of each of the at least two similar video frames that match the content of the second video, the video segment that matches the content of the second video is determined.

[0198] For each short video (i.e., the second video) related to the content of a film or television drama, based on the video segment that matches the content of the short video from the film or television drama, the video popularity value of the matching short video can be obtained, and the statistical units of the popularity value of the film or television drama within the video segment can be determined. The video popularity value is then accumulated into the unit popularity value of each statistical unit of the popularity value within the video segment.

[0199] In a specific application, for example, TV series A has aired 20 episodes, and there are 3000 related short videos. In this application, the frame vectors of these 20 episodes are indexed. Then, the frame feature sequence (i.e., multiple second video frame features) of each of the 3000 related short videos is extracted. Each frame feature (i.e., the second video frame feature) is used to query the corresponding original video in the index. If the frame feature most similar to the first frame of short video a is the first frame at 1000 seconds of episode 2 in TV series A, and the frame feature most similar to the last frame of short video a is the first frame at 1100 seconds of episode 2 in TV series A, then the 1000th second of episode 2 in TV series A is recorded as the start time, and the 1100th second of episode 2 in TV series A is recorded as the end time. That is, the video segment matching the content of short video a is from the 1000th second to the 1100th second of episode 2 in TV series A. Assuming short video 'a' has a video popularity value of 5000, then the cumulative unit popularity value (per frame or per second) of each popularity value statistical unit (from second 1000 to second 1100 of episode 2 of TV series A) is 5000. By iterating through all relevant short videos of TV series A, the cumulative popularity of each episode at each time point can be obtained, which is the unit popularity value of each popularity value statistical unit in the first video. This popularity is derived from the cumulative video popularity values ​​of the short videos and can be normalized as needed. It should be noted that since this application only needs to find highlight segments, the number of views of the short videos can be directly used as the video popularity value.

[0200] In practical applications, the highlight clipping section requires calculating the popularity of each episode and clipping highlight clips. Specifically, after obtaining the cumulative popularity of each episode of a TV series at each time point, a threshold can be set for it. A popularity threshold of T is defined as the statistical unit with a popularity value exceeding T. This results in multiple target popularity statistical units. A portion of these target popularity statistical units that are consecutive in time constitute a popularity statistical unit segment. For each popularity statistical unit segment, the corresponding segment of video footage, i.e., the popular clips, can be extracted from the TV series.

[0201] In a specific application, taking a frame as the unit of popularity score statistics, frames with a popularity score exceeding T are considered popular frames. This yields a series of popular frames. The time intervals of consecutive popular frames can be used as popularity score statistics unit segments. These segments can then be used to extract popular segments from movies and TV shows. Taking a second as the unit of popularity score statistics, the resulting popularity-time graphs of some movies and TV shows can be illustrated as follows: Figure 8 As shown, by Figure 8 It can be seen that the unit heat values ​​of the heat value statistical units in time segment 802, time segment 804, time segment 806, time segment 808 and time segment 810 are relatively high, so these time segments can be used as heat value statistical unit segments.

[0202] In a specific application, after extracting multiple popular clips, it is necessary to merge and filter these popular clips. Specifically, for each popular clip, if the time interval between the popular clip and its adjacent popular clips is less than a preset time interval threshold, the popular clip and its adjacent popular clips can be merged to obtain a merged video clip. Each merged video clip and the unmerged video clips are then used as multiple candidate video clips. Next, from these candidate video clips, excessive segments are filtered out according to a preset duration threshold to obtain the final highlight clip.

[0203] In a specific application, for example, suppose the targeted video segment is from 300 seconds to 320 seconds of episode 2, and the adjacent video segment is from 322 seconds to 350 seconds of episode 2. The calculated time interval between the segments is 2 seconds, which is less than the preset time interval threshold. The server will merge the targeted video segment with the adjacent video segment and, during the merging process, supplement the content from 320 seconds to 322 seconds of episode 2. In other words, the merged video segment includes the video content from 300 seconds to 350 seconds of episode 2.

[0204] In practical applications, during the highlight clip title generation part based on a large model, a pre-trained multimodal large model can be used to generate clip description information for the highlight clip. Specifically, this can be a clip title describing the plot of the highlight clip, facilitating downstream applications. The specific implementation involves inputting the generated highlight clip, its corresponding film / TV series second video description information, and appropriate prompts into the pre-trained multimodal large model, allowing the model to output clip description information that is relevant to the plot and consistent with the content of the highlight clip.

[0205] It should be noted that the key video segment determination method of this application can be applied to highlight segment material mining, producing highlight content for each episode of a film or television series, assisting in short video production, film and television work operation, and promotion. For example, the highlight content produced for a specific episode of a film or television series can be as follows: Figure 9 As shown, it includes multiple highlight clips, and each highlight clip has clip description information, which may include the name of the film or television series (e.g., Figure 9 The image shows a TV series, and the number of film and television episodes (e.g., [the number of episodes is missing]). Figure 9 The image shown is from episode 15, containing information such as film and television characters, actors, scene descriptions, and scene start times. Further details include... Figure 9 As shown, a highlight clip material library can be constructed based on the highlight clips produced by the key video clip determination method of this application, forming a query mechanism that allows for querying and use based on material titles, material identifiers, film and television drama names, etc.

[0206] It should be noted that the film and television highlight segment mining technology proposed in this application has a certain degree of universality. While maintaining the overall processing flow structure, other models can be used to replace the functions of the sub-modules in the solution. For example, replacing the CLIP method used in this application with frame feature calculation methods such as ResNet (Residual Network), Inception (a classic deep learning architecture mainly used for image recognition tasks), EfficientNet (an efficient convolutional neural network architecture), and MobileNet (a lightweight deep convolutional neural network architecture) can achieve similar results.

[0207] Furthermore, the technical solutions proposed in this application are not limited to film and television works, but can also be applied to other scenarios, such as live broadcast replays, sports events, game replays, etc.

[0208] The video playback method provided in this application embodiment can be applied to, for example, Figure 10 In the application environment shown, terminal 1002 plays a first video on the video playback interface. In response to a trigger event for a key video segment in the first video, the playback timeline 1004 of the first video is displayed on the video playback interface. The key video segment is determined based on the unit popularity value of each popularity value statistical unit in the first video. The unit popularity value of each popularity value statistical unit in the first video is obtained based on the video popularity values ​​of multiple second videos related to the content of the first video, and the video segments in the first video that match the multiple second videos. The key video segment is located at the start time of the segment corresponding to the key video segment on the playback timeline. Figure 10 (Identified by a small black circle), indicating a critical moment prompt 1006.

[0209] The terminal 1002 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses.

[0210] In one exemplary embodiment, such as Figure 11 As shown, a video playback method is provided. This embodiment illustrates the method applied to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes steps 1202 to 1206. Wherein:

[0211] Step 1102: Play the first video on the video playback interface.

[0212] Step 1104: In response to the time-prompt trigger event for the key video segment of the first video, display the playback timeline of the first video on the video playback interface; the key video segment is determined based on the unit popularity value of each popularity value statistical unit in the first video; the unit popularity value of each popularity value statistical unit in the first video is obtained based on the video popularity values ​​of multiple second videos related to the content of the first video, and the video segments in the first video that match each of the multiple second videos.

[0213] Step 1106: Display a key moment prompt at the start time of the segment corresponding to the key video segment on the playback timeline.

[0214] For example, the terminal can play a first video on the video playback interface. During the playback of the first video, in response to a time prompt trigger event for a key video segment of the first video, the terminal will display the playback timeline of the first video on the video playback interface, and display a key moment prompt at the start time of the segment corresponding to the key video segment on the playback timeline. Specifically, the key moment prompt can be the segment description information of the key video segment.

[0215] In practical applications, the real-time prompt trigger event can be triggered by the user watching the first video. That is, while the user is watching the first video, a trigger entry can be displayed on the video playback interface to prompt the user to view the key video segments of the first video. In response to the user's trigger operation on the trigger entry, the terminal will trigger the real-time prompt trigger event for the key video segments of the first video.

[0216] In practical applications, the real-time prompt event can be automatically triggered during the playback of the first video. Specifically, at the very beginning of playback or after a certain playback duration, the terminal can automatically trigger a real-time prompt event for key video segments of the first video. If the automatic trigger only occurs after a certain playback duration, this duration can be configured according to the actual application scenario. For example, the specific playback duration could be 5 minutes.

[0217] In specific applications, key video segments are pre-determined from the first video. The determination method for key video segments can be as follows: acquire multiple second videos related to the content of the first video; determine video segments that match each of the multiple second videos from the first video; determine the popularity value statistical units within each video segment of the first video; acquire the video popularity values ​​of each of the multiple second videos; accumulate the video popularity values ​​of each of the multiple second videos into the unit popularity value of the popularity value statistical unit of each video segment; and determine the key video segments from the first video based on the unit popularity value of each popularity value statistical unit in the first video.

[0218] The aforementioned video playback method, during the playback of the first video, responds to a time-based prompt trigger event by displaying the playback timeline of the first video. At the start time of the segment corresponding to the key video segment on the playback timeline, a key moment prompt is displayed. When determining the key video segment, multiple second videos related to the content of the first video are acquired. From the first video, video segments matching each of the multiple second videos are identified, and the video popularity values ​​of each of the multiple second videos are obtained. These video popularity values ​​are accumulated into the unit popularity value of each video segment's popularity value statistics unit. This allows for objective statistical analysis of the unit popularity values ​​of each popularity value statistics unit in the first video, thereby enabling the determination of the key video segment from the first video based on these unit popularity values. The entire process, by first matching the video content of the second videos to determine video segments and then using the video popularity values ​​of the second videos for popularity analysis, achieves accurate determination of the key video segment, improving the accuracy of key video segment location. Based on this, key video segments can be accurately located during the playback of the first video, thereby improving the accuracy of key video segment location.

[0219] In one exemplary embodiment, the video playback method further includes:

[0220] In response to a trigger action for a key moment prompt, the playback timeline is displayed to switch from the current playback position to the start time of the key video segment, and the key video segment is played.

[0221] For example, in response to a user's trigger action regarding a key moment prompt, the terminal displays the playback timeline switching from the current playback position to the start time of the key video segment, and plays the key video segment, allowing the user to directly watch the key video segment and improving the user's viewing experience. In specific applications, such as... Figure 12 As shown, taking the current playback time as 5 minutes and the starting time of the key video segment as 20 minutes as an example, in response to the user's trigger operation for the key moment prompt 1202, the terminal will display the playback timeline switching from 5 minutes to 10 minutes and play the key video segment.

[0222] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0223] Based on the same inventive concept, this application also provides a key video segment determination device for implementing the key video segment determination method described above, and a video playback device for implementing the video playback method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more key video segment determination devices and video playback devices provided below can be found in the limitations of the key video segment determination method and video playback method described above, and will not be repeated here.

[0224] In one exemplary embodiment, such as Figure 13 As shown, a key video segment determination device is provided, including: a video acquisition module 1302, a segment matching module 1304, a statistical unit determination module 1306, a heat value accumulation module 1308, and a segment determination module 1310, wherein:

[0225] The video acquisition module 1302 is used to acquire multiple second videos related to the content of the first video;

[0226] The segment matching module 1304 is used to determine, from the first video, video segments that match each of the multiple second videos;

[0227] The statistical unit determination module 1306 is used to determine the heat value statistical unit in each video segment of the first video.

[0228] The popularity value accumulation module 1308 is used to obtain the video popularity value of each of the multiple second videos and accumulate the video popularity values ​​of each of the multiple second videos into the unit popularity value of the popularity value statistics unit of each video segment.

[0229] The segment determination module 1310 is used to determine key video segments from the first video based on the unit heat value of each heat value statistical unit in the first video.

[0230] The aforementioned key video segment determination device acquires multiple second videos related to the content of a first video, identifies video segments from the first video that match each of the multiple second videos, determines the heat value statistical units within each video segment of the first video, acquires the video heat values ​​of each of the multiple second videos, and accumulates these values ​​into the unit heat value of each video segment's heat value statistical unit. This allows for objective statistical analysis of the unit heat values ​​of each heat value statistical unit in the first video using the video heat values ​​of the second videos. Therefore, key video segments can be determined from the first video based on the unit heat values ​​of each heat value statistical unit in the first video. The entire process, by first matching the video content of the second videos to determine video segments and then using the video heat values ​​of the second videos for heat analysis, enables accurate determination of key video segments and improves the accuracy of key video segment location.

[0231] In an exemplary embodiment, the video acquisition module is further configured to acquire multiple candidate videos related to the first video theme, extract multiple first video frames from the first video, extract multiple second video frames from each candidate video, perform video frame matching between the multiple second video frames of each candidate video and the multiple first video frames respectively, obtain the video frame matching result of each second video, and obtain the second video related to the content of the first video based on the video frame matching result of each candidate video.

[0232] In an exemplary embodiment, the video acquisition module is further configured to extract features from multiple first video frames to obtain multiple first video frame features, extract features from multiple second video frames of each candidate video to obtain multiple second video frame features, and perform feature matching between the multiple second video frame features of the candidate video and the multiple first video frame features to obtain the video frame matching result of the candidate video.

[0233] In an exemplary embodiment, the video acquisition module is further configured to perform border detection on each second video frame of the candidate video to obtain border detection results, determine the area where the video frame is located in the second video frame based on the border detection results, and extract features from the area where the video frame is located to obtain the features of the second video frame.

[0234] In an exemplary embodiment, the video acquisition module is further configured to acquire first video description information of the first video, and retrieve multiple candidate videos related to the theme of the first video based on the first video description information.

[0235] In an exemplary embodiment, the video acquisition module is further configured to acquire sample video description information of each of the multiple sample videos, perform correlation matching between the first video description information and the sample video description information of each of the multiple sample videos to obtain correlation matching results, and obtain multiple candidate videos related to the theme of the first video from the multiple sample videos based on the correlation matching results.

[0236] In an exemplary embodiment, the segment matching module is further configured to determine at least two similar video frames from the first video that match the content of each of the plurality of second videos, and to determine the video time of each of the at least two similar video frames, and for each second video, to determine a video segment that matches the content of the second video based on the video time of each of the at least two similar video frames that match the content of the second video.

[0237] In an exemplary embodiment, the segment determination module is further configured to filter multiple target popularity value statistical units from each popularity value statistical unit in the first video according to the unit popularity value of each popularity value statistical unit in the first video, determine at least one popularity value statistical unit segment from the multiple target popularity value statistical units, and determine key video segments from the first video based on the at least one popularity value statistical unit segment.

[0238] In an exemplary embodiment, the segment determination module is further configured to, when there are multiple heat value statistical unit segments, extract the segment video segments corresponding to the heat value statistical unit segments from the first video for each heat value statistical unit segment, integrate the segment content of multiple segment video segments to obtain multiple candidate video segments, and filter out key video segments from the multiple candidate video segments based on the segment duration of each of the multiple candidate video segments.

[0239] In an exemplary embodiment, the segment determination module is further configured to determine the segment time interval between the targeted segment video segment and adjacent segment video segments for each segment video segment, and merge the targeted segment video segment and adjacent segment video segments if the segment time interval is less than a preset time interval threshold to obtain a merged video segment, and use each merged video segment and unmerged segment video segments as multiple candidate video segments.

[0240] In one exemplary embodiment, the key video segment determination device further includes a segment processing module, which is used to acquire second video description information of the first video. The second video description information and the key video segment are input into a pre-trained video segment description model, and the pre-trained video segment description model is used to predict segment description information of the key video segment.

[0241] In one exemplary embodiment, such as Figure 14 As shown, a video playback device is provided, including: a video playback module 1402, a timeline display module 1404, and a prompt display module 1406, wherein:

[0242] The video playback module 1402 is used to play the first video on the video playback interface;

[0243] The timeline display module 1404 is used to display the playback timeline of the first video on the video playback interface in response to a time prompt trigger event for a key video segment of the first video; the key video segment is determined based on the unit popularity value of each popularity value statistical unit in the first video; the unit popularity value of each popularity value statistical unit in the first video is obtained based on the video popularity values ​​of multiple second videos related to the content of the first video, and the video segments in the first video that match each of the multiple second videos.

[0244] The prompt display module 1406 is used to display a key moment prompt at the start time of the segment corresponding to the key video segment on the playback timeline.

[0245] The aforementioned video playback device, during the playback of the first video, responds to a time-prompt trigger event by displaying the playback timeline of the first video. At the start time of the segment corresponding to the key video segment on the playback timeline, a key moment prompt is displayed. When determining the key video segment, multiple second videos related to the content of the first video are acquired. From the first video, video segments matching each of the multiple second videos are identified, and the video popularity values ​​of each of the multiple second videos are obtained. These video popularity values ​​are accumulated into the unit popularity value of each video segment's popularity value statistics unit. This allows for objective statistical analysis of the unit popularity values ​​of each popularity value statistics unit in the first video, thereby enabling the determination of the key video segment from the first video based on these unit popularity values. The entire process, by first matching the video content of the second videos to determine video segments and then using the video popularity values ​​of the second videos for popularity analysis, enables accurate determination of the key video segment, improving the accuracy of key video segment positioning. Based on this, key video segments can be accurately located during the playback of the first video, thereby improving the accuracy of key video segment location.

[0246] In one exemplary embodiment, the playback module is further configured to, in response to a trigger operation for a key moment prompt, display the start time of the key video segment where the playback timeline switches from the current playback position, and play the key video segment.

[0247] The modules in the aforementioned key video segment determination device and video playback device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0248] In one exemplary embodiment, a computer device is provided. This computer device can be a terminal or a server. Taking the computer device as a terminal as an example, its internal structure diagram can be as follows: Figure 15As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for determining key video segments and a video playback method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0249] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0250] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0251] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0252] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0253] It should be noted that the data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0254] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0255] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0256] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for determining key video segments, characterized in that, The method includes: Retrieve multiple second videos related to the content of the first video; From the first video, determine the video segments that match each of the plurality of second videos; Determine the heat value statistical unit within each video segment of the first video; Obtain the video popularity value of each of the plurality of second videos, and accumulate the video popularity values ​​of each of the plurality of second videos into the unit popularity value of the popularity value statistics unit of each of the video segments; Based on the unit popularity value of each popularity value statistical unit in the first video, key video segments are determined from the first video.

2. The method according to claim 1, characterized in that, The acquisition of multiple second videos related to the content of the first video includes: Obtain multiple candidate videos related to the first video topic; Multiple first video frames are extracted from the first video, and multiple second video frames are extracted from each of the candidate videos respectively; Each candidate video's multiple second video frames are matched with the multiple first video frames to obtain a video frame matching result for each candidate video. Based on the video frame matching results of each candidate video, a plurality of second videos related to the content of the first video are obtained.

3. The method according to claim 2, characterized in that, The step of performing video frame matching between each candidate video's plurality of second video frames and the plurality of first video frames to obtain a video frame matching result for each candidate video includes: Feature extraction is performed on the plurality of first video frames respectively to obtain the features of the plurality of first video frames; For each candidate video, feature extraction is performed on multiple second video frames of the candidate video to obtain multiple second video frame features; The multiple second video frame features of the candidate video are respectively matched with the multiple first video frame features to obtain the video frame matching result of the candidate video.

4. The method according to claim 3, characterized in that, The step of extracting features from multiple second video frames of the candidate video to obtain multiple second video frame features includes: For each second video frame of the candidate video, border detection is performed on the second video frame to obtain border detection results; Based on the border detection results, the region where the video frame is located in the second video frame is determined, and feature extraction is performed on the region where the video frame is located to obtain the features of the second video frame.

5. The method according to claim 2, characterized in that, The step of obtaining multiple candidate videos related to the first video topic includes: Obtain the first video description information of the first video; Based on the first video description information, multiple candidate videos related to the first video topic are retrieved.

6. The method according to claim 5, characterized in that, The step of retrieving multiple candidate videos related to the first video topic based on the first video description information includes: Obtain the sample video description information for each of the multiple sample videos; The first video description information is matched with the sample video description information of each of the plurality of sample videos to obtain the correlation matching result. Based on the correlation matching results, multiple candidate videos related to the first video theme are obtained from the multiple sample videos.

7. The method according to any one of claims 1 to 6, characterized in that, The step of determining the video segment that matches each of the plurality of second videos from the first video includes: From the first video, at least two similar video frames that match the content of each of the plurality of second videos are determined, and the video time of each of the at least two similar video frames is determined; For each of the second videos, a video segment matching the content of the second video is determined based on the video times of the at least two similar video frames that match the content of the second video.

8. The method according to any one of claims 1 to 6, characterized in that, The step of determining key video segments from the first video based on the unit popularity values ​​of each popularity value statistical unit in the first video includes: Based on the unit popularity value of each popularity value statistical unit in the first video, multiple target popularity value statistical units are selected from each popularity value statistical unit in the first video; From the plurality of target heat value statistical units, at least one heat value statistical unit segment is determined; Based on the at least one heat value statistical unit segment, key video segments are determined from the first video.

9. The method according to claim 8, characterized in that, The step of determining key video segments from the first video based on the at least one heat value statistical unit segment includes: When there are multiple heat value statistical unit segments, for each heat value statistical unit segment, the segment video segment corresponding to the heat value statistical unit segment is extracted from the first video; Multiple video segments are integrated to obtain multiple candidate video segments; Based on the duration of each of the multiple candidate video segments, key video segments are selected from the multiple candidate video segments.

10. The method according to claim 9, characterized in that, The process of integrating the content of multiple video segments yields multiple candidate video segments, including: For each video segment, determine the time interval between the targeted video segment and adjacent video segments; If the time interval between segments is less than a preset time interval threshold, the targeted video segment and the adjacent video segment are merged to obtain a merged video segment. Each merged video segment and each unmerged segment of video video are used as multiple candidate video segments.

11. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain the second video description information of the first video; The second video description information and the key video segment are input into a pre-trained video segment description model, and the segment description information of the key video segment is predicted by the pre-trained video segment description model.

12. A video playback method, characterized in that, The method includes: Play the first video on the video playback interface; In response to a time-prompt event triggered for a key video segment of the first video, the playback timeline of the first video is displayed on the video playback interface; the key video segment is determined based on the unit popularity value of each popularity value statistical unit in the first video; the unit popularity value of each popularity value statistical unit in the first video is obtained based on the video popularity values ​​of multiple second videos related to the content of the first video, and the video segments in the first video that match each of the multiple second videos; A key moment prompt is displayed at the start time of the segment corresponding to the key video segment on the playback timeline.

13. The method according to claim 12, characterized in that, The method further includes: In response to a trigger operation for the key moment prompt, the playback timeline is displayed to switch from the current playback position to the start time of the key video segment, and the key video segment is played.

14. A device for determining key video segments, characterized in that, The device includes: The video acquisition module is used to acquire multiple second videos related to the content of the first video; The segment matching module is used to determine, from the first video, video segments that match each of the plurality of second videos; The statistical unit determination module is used to determine the heat value statistical unit in each video segment of the first video; The popularity value accumulation module is used to obtain the video popularity value of each of the plurality of second videos and accumulate the video popularity values ​​of each of the plurality of second videos into the unit popularity value of the popularity value statistics unit of each video segment. The segment determination module is used to determine key video segments from the first video based on the unit popularity value of each popularity value statistical unit in the first video.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

Images