Viewing point generation method and device, electronic equipment and readable medium

Abstract

The invention discloses a viewpoint generation method and device, electronic equipment and a readable medium, and relates to the technical field of data processing. The method comprises the following steps: segmenting a to-be-processed video to obtain a plurality of video clips; for each video clip, generating summary information corresponding to the video clip according to the line text corresponding to the video clip and third-party summary information corresponding to the to-be-processed video; generating a story line corresponding to the to-be-processed video according to the third-party summary information corresponding to the to-be-processed video; matching the summary information corresponding to the plurality of video clips with a story line, and determining a target video clip matched with the story line; and generating a watching point of the to-be-processed video according to the summary information corresponding to the target video clip. According to the method, the summary information of the video clip and the story line of the to-be-processed video are utilized to automatically screen out the target summary information which is more suitable for the story plot, so that the watching points are generated based on the target summary information, the watching points can be generated efficiently in batches, and the efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and readable medium for generating an attractive point. Background Art

[0002] Currently, common online video applications all offer a video highlight feature. By providing corresponding video highlights for online videos, users can preview the highlights of the video in advance, making it easier for them to browse the video's highlights. However, current highlight features are often generated manually, for example, by operators watching the video, identifying and marking the highlights, and then summarizing the highlights. This manual method of generating video highlight features is time-consuming and labor-intensive, requiring significant human resources and resulting in low efficiency. Given the massive amount of videos available online, it is impossible to generate highlight features for every video. Summary of the Invention

[0003] To solve the above technical problems or at least partially solve the above technical problems, embodiments of the present invention provide a method, device, electronic device, and computer-readable medium for generating an aspect.

[0004] In a first aspect of an embodiment of the present invention, a method for generating an interesting point is provided, comprising:

[0005] Split the video to be processed into multiple video clips;

[0006] For each video segment, generating summary information corresponding to the video segment according to the dialogue text corresponding to the video segment and the third-party summary information corresponding to the video to be processed;

[0007] Generating a story line corresponding to the video to be processed according to the third-party summary information corresponding to the video to be processed;

[0008] matching the summary information corresponding to the plurality of video clips with the story line, and determining target summary information matching the story line;

[0009] Generate highlights of the video to be processed according to the target summary information.

[0010] In a second aspect of an embodiment of the present invention, a device for generating an interesting point is provided, including:

[0011] A segmentation module is used to segment the video to be processed to obtain multiple video segments;

[0012] A summary generation module is configured to generate summary information corresponding to each video segment based on the dialogue text corresponding to the video segment and the third-party summary information corresponding to the video to be processed;

[0013] A storyline generation module, configured to generate a storyline corresponding to the video to be processed based on the third-party summary information corresponding to the video to be processed;

[0014] a screening module, configured to match summary information corresponding to the plurality of video clips with the story line, and determine target summary information matching the story line;

[0015] A highlight generation module is used to generate the highlight of the video to be processed according to the target summary information.

[0016] In a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a processor, a communication interface, a memory, and a communication bus; wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the highlight generation method of any embodiment of the present invention when executing the program stored in the memory.

[0017] In a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the highlight generation method of any embodiment of the present invention are implemented.

[0018] One embodiment of the above invention has the following advantages or beneficial effects:

[0019] The method for generating highlights of an embodiment of the present invention divides a video to be processed into multiple video segments, and generates summary information corresponding to each video segment based on the dialogue text corresponding to each video segment and the third-party summary information corresponding to the video to be processed. This method can batch generate summary information corresponding to each video segment, and thus batch generate highlights of the video to be processed, thereby improving the efficiency of highlight generation and saving costs. Based on the third-party summary information corresponding to the video to be processed, a story line of the video to be processed is generated, and the summary information corresponding to each video segment is matched with the story line of the video to be processed to determine target summary information that matches the story line, thereby screening out target summary information that is more consistent with the storyline. Then, based on the target summary information, highlights of the video to be processed are generated, so that the generated highlights are consistent with the storyline and attract attention. This technical solution not only improves the efficiency of highlight generation and saves manpower, but also generates highlights that are consistent with the storyline and attract attention.

[0020] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0022] Figure 1A schematic diagram showing a flow chart of a method for generating highlights according to an embodiment of the present invention;

[0023] Figure 2 A schematic diagram showing a flow chart of a method for generating highlights according to another embodiment of the present invention;

[0024] Figure 3 A schematic diagram showing a flow chart of a method for generating highlights according to another embodiment of the present invention;

[0025] Figure 4 A schematic diagram showing a flow chart of a method for generating highlights according to another embodiment of the present invention;

[0026] Figure 5 A schematic structural diagram of a highlight generating device according to an embodiment of the present invention is shown;

[0027] Figure 6 A schematic structural diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0030] Figure 1 FIG. 1 is a flow chart showing a method for generating a highlight according to an embodiment of the present invention. Figure 1 As shown, the method includes steps S101-S105.

[0031] Step S101: Segment the video to be processed to obtain multiple video segments.

[0032] The video to be processed may be, for example, a TV series, a movie, a variety show, a documentary, or other types of videos. Taking a TV series as an example, the video to be processed may be a single episode of the TV series.

[0033] In some optional embodiments, the video to be processed can be segmented according to a fixed period, for example, the video to be processed can be segmented every N seconds, where N is a positive number greater than 0. In other optional embodiments, a frame of image can be extracted from the video to be processed every M frames, and the extracted image can be used as a key frame image. Then, the similarity of adjacent key frame images is calculated, and multiple similar key frame images form a video segment, and the start and end time of the video segment are used as the segmentation points for segmenting the video to be processed. Among them, as an optional example, the similarity of adjacent key frame images can be calculated based on the SSIM (Structural Similarity) algorithm, or the key frame images can be vectorized, and the similarity between the vectors of adjacent key frame images (such as cosine similarity) is calculated, and the similarity between the vectors is used as the similarity of the key frame images.

[0034] The purpose of the embodiments of the present invention is to generate compelling, worthwhile highlights based on the data of the video being processed (e.g., the dialogue and third-party summary information of the video being processed, as described below). Directly analyzing the entire video being processed would result in a large granularity and potentially inaccurate highlights. Therefore, this embodiment divides the video being processed into multiple segments and analyzes them in units of segments. This results in a smaller granularity and minute-by-minute summary information for the segments, resulting in more accurate highlights.

[0035] Step S102: for each video segment, generating summary information corresponding to the video segment according to the dialogue text corresponding to the video segment and the third-party summary information corresponding to the video to be processed.

[0036] The third-party summary information corresponding to the video to be processed may include, for example, a summary of the video to be processed, which records the important content of the video to be processed. For example, if the video to be processed is a single episode of a TV series, the third-party summary information corresponding to the video to be processed may include, for example, a synopsis of the TV series, a plot summary, and the episode plot.

[0037] In some optional embodiments, summary information for video clips can be obtained based on a machine learning algorithm, such as a supervised learning algorithm. For example, a large number of video clips, their corresponding dialogue texts, and summary information corresponding to manually annotated video clips can be obtained. Third-party summary information of the complete videos corresponding to the video clips, the dialogue texts of the video clips, and the summary information corresponding to the video clips can be used as samples for training to obtain a summary generation model. This summary generation model is used to generate summary information for each video clip in the video to be processed.

[0038] In other optional embodiments, a pre-trained language model (Generative Pre-Trained Transformer, GPT) and prompts can be used to generate summaries of video clips. Pre-trained language models leverage larger amounts of text data and deeper neural network models to learn richer semantic representations of text. Furthermore, they break down the barriers between various natural language processing tasks, making it easier to build a natural language processing model for a specific task without having to understand extensive task context. Instead, pre-trained language models can be applied based on the input and output of the task to complete the task. Prompt learning is a method that utilizes pre-trained language models to complete different tasks without requiring retraining or fine-tuning. Using prompt learning reduces model training costs and data requirements. The core of prompt learning is designing appropriate prompts. Prompts are a specific input format used to guide the pre-trained language model to produce the desired output. These prompts can be a simple question, a longer text, a set of instructions, a description, or any other information that can provide the context and theme required for generation. When generating content, the pre-trained language model first processes the prompts and then generates output based on its understanding. For example, in an embodiment of the present application, prompt words for generating video clip summaries can be pre-constructed, and summary information of each video clip can be obtained using the prompt words and a pre-trained language model.

[0039] Step S103: Generate a story line corresponding to the video to be processed according to the third-party summary information corresponding to the video to be processed.

[0040] The storyline refers to the main plot line of the story told by the video to be processed, a linear narrative leading to a certain ending.

[0041] In some optional embodiments, the story line of the video can be obtained based on a machine learning algorithm, such as a supervised learning algorithm. For example, third-party summary information and manually annotated story lines of a large number of videos can be obtained, and the third-party summary information and manually annotated story lines of a large number of videos can be used as samples for training to obtain a story line generation model. The story line of the video to be processed is generated using the story line generation model. In other optional embodiments, a pre-trained language model (Generative Pre-Trained Transformer, GPT for short) and prompts can be used to generate summaries of video clips. For example, prompt words for generating story lines are pre-constructed, and the story line of the video to be processed is obtained using the prompt words and the pre-trained language model. Optionally, the prompt words for generating story lines can be designed according to the structure of roles, tasks, requirements, and prompts. As an optional example, the prompt words for generating story lines include:

[0042] You are an intelligent assistant that can summarize the story of a video based on the video lines and sort out at least one storyline of the story according to the following logic: each specific event and development in the story is a storyline, and a series of logically related and causally related storylines constitute a storyline. A story is composed of one or more storylines. The content of each storyline output is as detailed as possible.

[0043] You can determine whether two storylines belong to the same storyline by the following three points:

[0044] (1) Character consistency: The characters in the storyline should be the same or related;

[0045] (2) Plot coherence: If there is an obvious causal relationship or logical connection between two plots, then they may belong to the same story line;

[0046] (3) Overall coordination: From a macro perspective, the two plots can be reasonably combined to form a story with a complete narrative structure, rather than independent fragments, which may belong to the same story line.

[0047] Example of storyline output format:

[0048] {

[0049] "Storyline 1": "Contents of Storyline 1",

[0050] "Storyline 2": "Contents of Storyline 2",

[0051] ...}.

[0052] Step S104: matching the summary information corresponding to the plurality of video clips with the story line, and determining target summary information matching the story line.

[0053] The target summary information that matches the storyline is the summary information that is more relevant to the storyline and more worth watching among the summary information corresponding to all video clips.

[0054] In some optional embodiments, the semantics of the summary information corresponding to each video clip and the semantics of the story line can be identified, and the target summary information that matches the story line can be determined based on the semantics of each summary information and the semantics of the story line. In other optional embodiments, a pre-trained language model (Generative Pre-Trained Transformer, GPT for short) and prompts can be used to generate summaries of video clips. For example, prompts for matching the summary information corresponding to the video clips with the story line are pre-constructed, and the target summary information of the video to be processed is obtained using the prompts and the pre-trained language model. Optionally, the prompts for matching the summary information corresponding to the video clips with the story line can be designed according to the structure of roles, tasks, requirements, and prompts. As an optional example, the prompts for matching the summary information corresponding to the video clips with the story line include:

[0055] You are an intelligent assistant. Consider a TV series episode with several video clips (listed, for example, numbered 1 to K). You understand the summary information for each clip and filter out those that match a given storyline based on the following logic: a video clip is considered to match a storyline if it occurs within it or if a plot segment belongs to it.

[0056] Step S105: generating highlights of the video to be processed according to the target summary information.

[0057] Among them, the highlights are the most important plot summary of the video to be processed, which refers to the content that is eye-catching or worth watching. In an optional embodiment, the highlights of the video to be processed can be generated using a pre-trained language model (Generative Pre-Trained Transformer, abbreviated as GPT) and prompts. For example, a prompt word for generating highlights is pre-constructed, and the prompt word and the pre-trained language model are used to generate highlights based on the target summary information. Optionally, the prompt word for generating highlights can be designed according to the structure of the task and requirements. As an optional example, the prompt word for generating highlights includes:

[0058] Based on the input target summary information, extract the most important plot and summarize it within 16 Chinese characters. Do not abbreviate people's names arbitrarily unless the target summary information contains abbreviations. Please ensure that the sentences are smooth.

[0059] The method for generating highlights of an embodiment of the present invention divides a video to be processed into multiple video segments. Based on the dialogue text corresponding to the video segments and the third-party summary information corresponding to the video to be processed, summary information corresponding to each video segment can be generated in batches, thereby generating highlights for the video to be processed in batches. Based on the third-party summary information corresponding to the video to be processed, the story line of the video to be processed is generated. The summary information corresponding to each video segment is matched with the story line of the video to be processed, and target summary information matching the story line is determined to screen out target summary information that is more consistent with the storyline. Then, the highlights of the video to be processed are generated based on the target summary information, so that the generated highlights are consistent with the storyline and eye-catching. This technical solution can generate summary information corresponding to each video segment in batches, improves generation efficiency, and saves costs.

[0060] In an optional embodiment, the method for generating highlights provided by an embodiment of the present invention includes, in obtaining highlights, the following steps: filtering out meaningless highlights from the highlights based on the semantic information of the highlights. For example, highlights that contain a person, a certain person, or an unclear reference word can be determined as meaningless highlights. For example, highlight 1 is "Two characters agree on an action plan to retreat to Qianxi", and the unclear reference word "role" exists in the highlight, so the highlight is determined to be a meaningless highlight. Show point 2 is "Someone suggests that another person stand in a certain position, indicating that this can win the typhoon", and the unclear reference word "role" exists in the highlight, so the highlight is determined to be a meaningless highlight.

[0061] In some optional embodiments, whether the highlights are meaningful can be judged based on preset judgment rules. For example, if there are preset meaningless keywords in the highlights, it is determined to be a meaningless highlight. In other optional embodiments, whether the highlights are meaningful can also be judged based on the use of a pre-trained language model (Generative Pre-Trained Transformer, GPT for short) and prompts. Optionally, the prompt words for judging whether the highlights are meaningful can be designed according to the structure of roles, tasks, requirements, and examples. As an optional example, the prompt words for judging whether the highlights are meaningful include: assuming that you are the editor-in-chief of a film or TV series, you need to judge the paragraph summary and whether the sentence is meaningful. Note: When a sentence contains unclear references to a person, a certain person, several roles, characters, etc., it is judged to be meaningless. The output format is: meaningful / meaningless. You can refer to the following examples:

[0062] Sentence 1: The two characters agreed on a plan of action to retreat to western Guizhou;

[0063] Output: meaningless.

[0064] Sentence 2: Zhang xx and Li xx are looking for clues at location xxx;

[0065] Output: Makes sense.

[0066] The embodiment of the present invention improves the quality of highlights and enhances the playback experience by screening out meaningless highlights.

[0067] In an optional embodiment, after obtaining the viewing points, the method further includes: performing deduplication processing on the viewing points based on the similarity between the viewing points. If the similarity between the viewing points is greater than a preset threshold, one of the viewing points is deleted. In an optional embodiment, the viewing points can be vectorized to determine the vectors corresponding to the viewing points; the similarity between the vectors is calculated, and the similarity between the vectors is used as the similarity between the viewing points. Viewpoints with a similarity greater than a preset threshold (e.g., 0.8) are regarded as duplicate viewing points, and duplicate viewing points are deduplicated.

[0068] As an optional example, the highlights may be vectorized according to the vector model BGE (Bagof Global Features).

[0069] The embodiment of the present invention performs deduplication processing on viewing points, thereby deleting duplicate viewing points, improving the quality of viewing points, and enhancing the playback experience.

[0070] In an optional embodiment, duplicate viewing points can be deduplicated based on the distance between the duplicate viewing points and adjacent viewing points, with viewing points that are closer to adjacent viewing points being deleted. For example, if a first viewing point and a second viewing point are duplicate viewing points, a first distance between the first viewing point and its adjacent viewing point and a second distance between the second viewing point and its adjacent viewing point are determined, respectively. If the first distance is less than the second distance, the first viewing point is removed; if the second distance is less than the first distance, the second viewing point is removed; and if the first distance is equal to the second distance, either the first viewing point or the second viewing point can be removed.

[0071] Figure 2 FIG2 shows a flow chart of a method for generating highlights of another embodiment of the present invention. This embodiment generates highlights of a video to be processed based on a pre-trained language model and prompts. Figure 2 As shown, the method includes:

[0072] Step S201: Segment the video to be processed to obtain multiple video segments.

[0073] For example, if the video being processed is a single episode of a TV series, a frame is extracted every several frames (e.g., 5 frames) and used as a keyframe image. The similarity between adjacent keyframe images is then calculated, and multiple similar keyframe images form a video segment. The number of keyframe images in each video segment is not fixed.

[0074] Step S202: Inputting the dialogue text corresponding to the video clip, the third-party summary information corresponding to the video to be processed, and the pre-constructed first prompt word as first input data into the pre-trained language model to obtain first output data of the pre-trained language model, where the first output data is the summary information corresponding to the video clip.

[0075] Step S203: inputting the third-party summary information and the pre-constructed second prompt word as second input data into the pre-trained language model to obtain second output data of the pre-trained language model, where the second output data is the story line corresponding to the video to be processed.

[0076] Step S204: The summary information, story lines, and pre-constructed third prompt words corresponding to the multiple video clips are input as third input data into the pre-trained language model to obtain third output data of the pre-trained language model, where the third output data is target summary information matching the story lines.

[0077] Step S205: inputting the target summary information and the pre-constructed fourth prompt word as fourth input data into the pre-trained language model to obtain fourth output data of the pre-trained language model, where the fourth output data is the highlights of the video to be processed.

[0078] Among them, the pre-trained language models in steps S202-S205 can be the same, for example, they can be GPT-4, baichuan2-13B, Yi-34B and other models.

[0079] As an optional example, the first prompt word could be: Use narrative sentences in the summary, and omit greetings, small talk, and other meaningless content; Note that the unknown character in the dialogue text may refer to multiple characters; The summary should try to include all key plot points related to the dialogue; The summary cannot copy the dialogue.

[0080] As an optional example, the second prompt word may be:

[0081] You are an intelligent assistant that can summarize the story of a video based on the video lines and sort out at least one storyline of the story according to the following logic: each specific event and development in the story is a storyline, and a series of logically related and causally related storylines constitute a storyline. A story is composed of one or more storylines. The content of each storyline output is as detailed as possible.

[0082] You can determine whether two storylines belong to the same storyline by the following three points:

[0083] (1) Character consistency: The characters in the storyline should be the same or related;

[0084] (2) Plot coherence: If there is an obvious causal relationship or logical connection between two plots, then they may belong to the same story line;

[0085] (3) Overall coordination: From a macro perspective, the two plots can be reasonably combined to form a story with a complete narrative structure, rather than independent fragments, which may belong to the same story line.

[0086] Example of storyline output format:

[0087] {

[0088] "Storyline 1": "Contents of Storyline 1",

[0089] "Storyline 2": "Contents of Storyline 2",

[0090] ...}.

[0091] As an optional example, the third prompt word could be: "You are an intelligent assistant." A certain episode of a TV series has several video clips (given in a list, for example, numbered 1 to K). A video clip that occurs within a certain storyline or a plot clip that belongs to a certain storyline is considered to match the storyline, and the summary information of the video clip is the target summary information.

[0092] As an optional example, the fourth prompt word can be: Based on the target summary information, extract the most important plot and summarize it within 16 Chinese characters; do not abbreviate people's names at will, unless the abbreviation is included in the target summary information, and please ensure that the sentence is smooth.

[0093] The highlight generation method of an embodiment of the present invention generates summary information of video clips in batches through a pre-trained language model and prompt, thereby generating highlights in batches more efficiently without the need to retrain or fine-tune the pre-trained language model, thereby reducing the training cost and data requirements of the model.

[0094] Figure 3 FIG. 1 is a flow chart showing a method for generating a highlight according to another embodiment of the present invention. Figure 3 As shown, the method includes:

[0095] Step S301: Segment the video to be processed to obtain multiple video segments.

[0096] Step S302: For each video clip, the dialogue text corresponding to the video clip, the third-party summary information corresponding to the video to be processed, and the pre-constructed first prompt word are input as first input data into a pre-trained language model to obtain first output data of the pre-trained language model, where the first output data is the summary information corresponding to the video clip.

[0097] Step S303: inputting the third-party summary information and the pre-constructed second prompt word as second input data into the pre-trained language model to obtain second output data of the pre-trained language model, where the second output data is the story line corresponding to the video to be processed.

[0098] Step S304: Obtain importance information corresponding to the video to be processed. The importance information is used to represent the importance of each frame image in the video to be processed.

[0099] Step S305: Filter important segments that meet preset conditions from multiple video segments according to the importance information.

[0100] Step S306: If the number of important segments is greater than or equal to the index value, the summary information of the important segments is used as the target summary information.

[0101] Step S307: If the number of important segments is less than the index value, the summary information corresponding to the multiple video segments is matched with the story line, the video segments matching the story line are determined, and the summary information of the important segments and the video segments matching the story line are used as the target summary information.

[0102] Step S308: Inputting the summary information corresponding to the target video clip and the pre-constructed fourth prompt word as fourth input data into the pre-trained language model to obtain fourth output data of the pre-trained language model, where the fourth output data is the highlights of the video to be processed.

[0103] Among them, steps S301-S303 and S308 are Figure 2 The embodiments shown are the same and will not be described again to avoid repetition.

[0104] In step S304, the video to be processed can be a video that is about to be released, or it can be a video that has been released for some time. For example, if the video to be processed is a video that has been released for some time, after the video is released, users watch the video to collect user viewing information, such as the viewing time and number of views of each video clip or frame. The longer the viewing time or the more views, the more eye-catching and exciting the video clip. Then, based on the user's viewing information, importance information for the video to be processed is generated. For example, a mapping function between importance information and viewing information is pre-designed, and the importance information of the video to be processed is determined according to the preset mapping function.

[0105] For step S305, the importance of each video segment can be determined based on the importance of each frame of the video to be processed (for example, the sum of the importance of all images in the video segment is used as the importance of the video segment), and then the video segment with an importance greater than a preset value is used as the important segment.

[0106] In steps S306-307, if the number of important segments is greater than or equal to an index value, such as 5, the summary information of the important segments is used as the target summary information, i.e., the important summary information. If the number of important segments is less than the index value, the summary information of each video segment is matched with the storyline of the video to be processed to determine the video segments that match the storyline. The video segments that match the storyline are more relevant to the plot and are more worthy of attention. The summary information of the important segments and the video segments that match the storyline is then used as the target summary information.

[0107] The highlight generation method of an embodiment of the present invention generates summary information of video clips in batches through a pre-trained language model and prompts, thereby more efficiently generating highlights in batches without the need to retrain or fine-tune the pre-trained language model, reducing the training cost and data requirements of the model; then, based on the story line and importance information of the video to be processed, important target summary information that is more relevant to the storyline is screened out from the summary information of all video clips, and the highlights generated based on the target summary information are more worthy of attention and more exciting.

[0108] Figure 4 FIG. 1 is a flow chart showing a method for generating a highlight according to another embodiment of the present invention. Figure 4 As shown, the method includes:

[0109] Step S401: Segment the video to be processed to obtain multiple video segments;

[0110] Step S402: determining the similarity between the two adjacent video clips based on the dialogue texts corresponding to the two adjacent video clips;

[0111] Step S403: If the similarity between the two adjacent video segments is greater than or equal to a preset second threshold, the two adjacent video segments are merged; if the similarity between the two adjacent video segments is less than the preset second threshold, the two adjacent video segments are not merged;

[0112] Step S404: For each video clip, the dialogue text corresponding to the video clip, the third-party summary information corresponding to the video to be processed, and the pre-constructed first prompt word are input as first input data into a pre-trained language model to obtain first output data of the pre-trained language model, where the first output data is the summary information corresponding to the video clip.

[0113] Step S405: Input the third-party summary information and the pre-constructed second prompt word as second input data into a pre-trained language model to obtain second output data of the pre-trained language model, where the second output data is the story line corresponding to the video to be processed.

[0114] Step S406: Obtaining importance information corresponding to the video to be processed, where the importance information is used to represent the importance of each frame of the video to be processed;

[0115] Step S407: Filtering important segments that meet preset conditions from the multiple video segments based on the importance information;

[0116] Step S408: When the number of the important segments is greater than or equal to the index value, the summary information of the important segments is used as the target summary information;

[0117] Step S409: When the number of the important segments is less than the index value, the summary information corresponding to the plurality of video segments is matched with the story line, the video segments matching the story line are determined, and the summary information of the important segments and the video segments matching the story line are used as target summary information.

[0118] Step S410: Inputting the target summary information and the pre-constructed fourth prompt word as fourth input data into a pre-trained language model to obtain fourth output data of the pre-trained language model, where the fourth output data is the highlights of the video to be processed.

[0119] Step S411: inputting the viewing point and the pre-constructed fifth prompt word as fifth input data into a pre-trained language model to obtain fifth output data of the pre-trained language model, wherein the fifth output data is a meaningless viewing point.

[0120] Step S412: performing vectorization processing on the viewing point to determine the vector corresponding to the viewing point;

[0121] Step S413: Calculate the similarity between the vectors corresponding to the viewing points, use the similarity between the vectors as the similarity between the viewing points, consider the viewing points with similarity greater than a preset first threshold as duplicate viewing points, and perform deduplication processing on the duplicate viewing points.

[0122] Among them, steps S404-S413 can refer to Figure 1-Figure 3 To avoid repetition, the illustrated embodiments will not be described again here.

[0123] In an optional embodiment, the video to be processed can be segmented every N seconds to obtain video segments, and the duration of each video segment is N seconds. If a scene in the video to be processed, such as a scene of a conversation between two characters, lasts for several minutes, the video of the scene will be cut into multiple video segments, and the similarity between adjacent video segments is relatively large. In response to this situation, this embodiment determines the similarity of the adjacent video segments based on the dialogue texts of the two adjacent video segments, and determines whether the two adjacent video segments belong to the same scene based on the similarity of the two adjacent videos. For example, the dialogue texts of the two adjacent video segments are input into a pre-trained language model, the semantics of the dialogue texts are identified by the pre-trained language model, the correlation between the dialogue texts is determined, and then the similarity of the two adjacent video segments is determined. If the similarity of the two adjacent video segments is greater than or equal to the second threshold, the adjacent video segments are merged, otherwise they are not merged.

[0124] The highlight generation method of the embodiment of the present invention first divides the video to be processed into multiple video segments, and merges the segments using a pre-trained language model based on the dialogue text of each video segment, thereby ensuring the segmentation effect and preventing the video to be processed from being fragmented; using the pre-trained language model to batch generate summary information of the merged video segments and the story line of the video to be processed, and combining the story line and importance information of the video to be processed to select video segments that are more in line with the storyline, and based on the summary information of the video segments that are more in line with the storyline, generate highlights for the video to be processed, so that the generated highlights are more in line with the storyline, more eye-catching, and more exciting, and use the pre-trained language model to remove meaningless highlights and deduplicate the highlights, thereby improving the quality of the highlights and enhancing the viewing experience. In this technical solution, there is no need to retrain or fine-tune the pre-trained language model, which reduces the training cost and data requirements of the model, improves the efficiency of highlight generation, and effectively saves labor costs.

[0125] Figure 5 FIG. 1 shows a schematic diagram of the structure of a highlight generating device according to an embodiment of the present invention. Figure 5 As shown, the highlight generating device 500 includes:

[0126] The segmentation module 501 is used to segment the video to be processed to obtain multiple video segments;

[0127] A summary generation module 502 is configured to generate summary information corresponding to each video segment based on the dialogue text corresponding to the video segment and the third-party summary information corresponding to the video to be processed;

[0128] A storyline generating module 503 is configured to generate a storyline corresponding to the video to be processed based on the third-party summary information corresponding to the video to be processed;

[0129] A screening module 504 is configured to match summary information corresponding to the plurality of video clips with the story line, and determine target summary information matching the story line;

[0130] The highlight generation module 505 is configured to generate the highlight of the video to be processed according to the target summary information.

[0131] The device for generating highlights provided by an embodiment of the present invention divides a video to be processed into multiple video segments. Based on the corresponding dialogue text of the video segments and the third-party summary information corresponding to the video to be processed, it can batch-generate summary information corresponding to each video segment, thereby batch-generating highlights for the video to be processed. Based on the third-party summary information corresponding to the video to be processed, it generates a storyline for the video to be processed. The summary information corresponding to each video segment is matched with the storyline corresponding to the video to be processed, and target summary information matching the storyline is determined to select target summary information that better fits the storyline. Then, based on the target summary information, highlights for the video to be processed are generated, ensuring that the generated highlights fit the storyline and are eye-catching. This technical solution can batch-generate summary information corresponding to each video segment, improving generation efficiency and saving costs.

[0132] In an optional embodiment, the summary generation module is further configured to: input the dialogue text corresponding to the video clip, the third-party summary information corresponding to the video to be processed, and the pre-constructed first prompt word as first input data into a pre-trained language model, and obtain first output data of the pre-trained language model, where the first output data is the summary information corresponding to the video clip.

[0133] In an optional embodiment, the storyline generation module is further configured to: input the third-party summary information and the pre-constructed second prompt word as second input data into a pre-trained language model to obtain second output data of the pre-trained language model, where the second output data is the storyline corresponding to the video to be processed.

[0134] In an optional embodiment, the screening module is further configured to: input summary information corresponding to the plurality of video clips, the story line, and a pre-constructed third prompt word as third input data into a pre-trained language model, and obtain third output data of the pre-trained language model, wherein the third output data is target summary information matching the story line.

[0135] In an optional embodiment, the highlight generation module is further used to: input the target summary information and the pre-constructed fourth prompt word as fourth input data into a pre-trained language model to obtain fourth output data of the pre-trained language model, and the fourth output data is the highlight of the video to be processed.

[0136] In an optional embodiment, the apparatus further includes a screening module configured to screen out meaningless viewing points from the viewing points based on semantic information of the viewing points.

[0137] In an optional embodiment, the screening module is further used to: input the highlights and the pre-constructed fifth prompt word as fifth input data into a pre-trained language model to obtain fifth output data of the pre-trained language model, where the fifth output data is a meaningless highlight.

[0138] In an optional embodiment, the device also includes a deduplication module for vectorizing the viewing points to determine the vectors corresponding to the viewing points; calculating the similarity between the vectors corresponding to the viewing points, using the similarity between the vectors as the similarity between the viewing points, and treating the viewing points whose similarity is greater than a preset first threshold as duplicate viewing points, and performing deduplication processing on the duplicate viewing points.

[0139] In an optional embodiment, the deduplication module is further configured to:

[0140] If the first viewpoint and the second viewpoint are repeated viewpoints, determine the first distance between the first viewpoint and its adjacent viewpoint, and the second distance between the second viewpoint and its adjacent viewpoint; if the first distance is less than the second distance, remove the first viewpoint; if the second distance is less than the first distance, remove the second viewpoint; if the first distance is equal to the second distance, remove the first viewpoint or the second viewpoint.

[0141] In an optional embodiment, the screening module is further used to: obtain importance information corresponding to the video to be processed, the importance information being used to characterize the importance of each frame image in the video to be processed; based on the importance information, screen important segments that meet preset conditions from the multiple video segments; if the number of the important segments is greater than or equal to the index value, use the summary information of the important segments as target summary information; if the number of the important segments is less than the index value, match the summary information corresponding to the multiple video segments with the story line, determine the target video segment that matches the story line, and use the important segment and the summary information of the target video segment as target summary information.

[0142] In an optional embodiment, the device further includes a merging module for determining the similarity between the two adjacent video clips based on the dialogue texts corresponding to the two adjacent video clips; and merging the two adjacent video clips when the similarity between the two adjacent video clips is greater than or equal to a preset second threshold.

[0143] The above device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0144] The embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0145] Memory 603, used for storing computer programs;

[0146] The processor 601 is configured to execute the program stored in the memory 603 by performing the following steps:

[0147] Split the video to be processed into multiple video clips;

[0148] For each video segment, generating summary information corresponding to the video segment according to the dialogue text corresponding to the video segment and the third-party summary information corresponding to the video to be processed;

[0149] Generating a story line corresponding to the video to be processed according to the third-party summary information corresponding to the video to be processed;

[0150] matching the summary information corresponding to the plurality of video clips with the story line, and determining target summary information matching the story line;

[0151] Based on the target summary information, highlights of the video to be processed are generated. The communication bus mentioned in the terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0152] The communication interface is used for communication between the above terminal and other devices.

[0153] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0154] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0155] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions that, when executed on a computer, enable the computer to execute any of the methods for generating highlights described in the above embodiments.

[0156] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the method for generating highlights described in any one of the above embodiments.

[0157] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0158] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0159] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A method for generating highlights, characterized in that: The method comprises: Splitting the video to be processed to obtain multiple video clips; For each video segment, generating summary information corresponding to the video segment according to the dialogue text corresponding to the video segment and the third-party summary information corresponding to the video to be processed; Generating a story line corresponding to the video to be processed according to the third-party summary information corresponding to the video to be processed; Matching summary information corresponding to the plurality of video clips with the story line, and determining target summary information matching the story line; The highlights of the video to be processed are generated according to the target summary information.

2. The method according to claim 1, characterized in that The step of generating summary information corresponding to the video clip according to the dialogue text corresponding to the video clip and the third-party summary information corresponding to the video to be processed includes: The dialogue text corresponding to the video clip, the third-party summary information corresponding to the video to be processed, and the pre-constructed first prompt word are input into a pre-trained language model as first input data to obtain first output data of the pre-trained language model, where the first output data is the summary information corresponding to the video clip.

3. The method according to claim 1, characterized in that Generating a story line corresponding to the video to be processed according to the third-party summary information corresponding to the video to be processed includes: The third-party summary information and the pre-constructed second prompt word are input into a pre-trained language model as second input data to obtain second output data of the pre-trained language model, where the second output data is a story line corresponding to the video to be processed.

4. The method according to claim 1, characterized in that: The step of matching the summary information corresponding to the plurality of video clips with the story line to determine target summary information matching the story line includes: The summary information corresponding to the plurality of video clips, the story line and the pre-constructed third prompt word are input as third input data into a pre-trained language model to obtain third output data of the pre-trained language model, where the third output data is target summary information matching the story line.

5. The method according to claim 1, characterized in that Generating the highlights of the video to be processed according to the target summary information includes: The target summary information and the pre-constructed fourth prompt word are input into a pre-trained language model as fourth input data to obtain fourth output data of the pre-trained language model, where the fourth output data is the highlights of the video to be processed.

6. The method according to claim 1, characterized in that After obtaining the highlights, the method further includes: Meaningless highlights are filtered out from the highlights according to the semantic information of the highlights.

7. The method according to claim 6, characterized in that The step of filtering out meaningless highlights from the highlights according to the semantic information of the highlights includes: The highlight and the pre-constructed fifth prompt word are input into a pre-trained language model as fifth input data to obtain fifth output data of the pre-trained language model, where the fifth output data is a meaningless highlight.

8. The method according to claim 1, characterized in that After obtaining the highlights, the method further includes: Performing vector processing on the viewing point to determine a vector corresponding to the viewing point; The similarity between the vectors corresponding to the viewing points is calculated, the similarity between the vectors is used as the similarity between the viewing points, the viewing points whose similarity is greater than a preset first threshold are used as duplicate viewing points, and the duplicate viewing points are deduplicated.

9. The method according to claim 8, characterized in that Deduplication processing is performed on the repeated highlights, including: If the first viewing point and the second viewing point are repeated viewing points, determining a first distance between the first viewing point and an adjacent viewing point, and a second distance between the second viewing point and an adjacent viewing point; When the first distance is smaller than the second distance, the first viewing point is removed; When the second distance is smaller than the first distance, the second viewing point is removed; When the first distance is equal to the second distance, the first viewing point or the second viewing point is removed.

10. The method according to any one of claims 1 to 9, characterized in that: The step of matching the summary information corresponding to the plurality of video clips with the story line to determine target summary information matching the story line includes: Obtaining importance information corresponding to the video to be processed, where the importance information is used to represent the importance of each frame of the video to be processed; According to the importance information, selecting important segments that meet preset conditions from the multiple video segments; When the number of the important segments is greater than or equal to the index value, taking the summary information of the important segments as the target summary information; When the number of the important segments is less than the index value, the summary information corresponding to the plurality of video segments is matched with the story line, a target video segment matching the story line is determined, and the important segments and the summary information of the target video segment are used as target summary information.

11. The method according to claim 1, characterized in that: Before generating summary information corresponding to the video clip, the method further includes: Determining the similarity between the two adjacent video clips according to the dialogue texts corresponding to the two adjacent video clips; When the similarity between the two adjacent video segments is greater than or equal to a preset second threshold, the two adjacent video segments are merged.

12. A device for generating highlights, characterized in that: include: A segmentation module is used to segment the video to be processed to obtain multiple video segments; A summary generation module, for generating summary information corresponding to each video segment according to the dialogue text corresponding to the video segment and the third-party summary information corresponding to the video to be processed; A storyline generation module, used to generate a storyline corresponding to the video to be processed according to the third-party summary information corresponding to the video to be processed; A screening module, used for matching summary information corresponding to the plurality of video clips with the story line, and determining target summary information matching the story line; A highlight generation module is used to generate the highlight of the video to be processed according to the target summary information.

13. An electronic device, characterized in that: including a processor, a communication interface, a memory and a communication bus; Wherein, the processor, the communication interface, and the memory communicate with each other via a communication bus; The memory is used to store computer programs; The processor is used to implement the method described in any one of claims 1-11 when executing the program stored in the memory.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images