Method, device, electronic device and storage medium for generating video summary

Abstract

The present application provides a method, device, electronic device and storage medium for generating a video summary. The method may include the following steps: determining the characteristics of a video segment, where the video segment is obtained by segmenting the video; determining the characteristics of a video image frame in the video segment, where the number of video image frames in the video segment is multiple; determining the characteristics of attention information, where the attention information is used to characterize the attention paid to the video in different dimensions; and generating a video summary of the video using the characteristics of the video segment, the characteristics of the video image frame in the video segment and the characteristics of the attention information. According to an embodiment of the present application, the adaptive generation of video summaries for content preferred by different users is achieved through attention information. The same set of models is implemented to meet the personalized needs of different users.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method and device for generating a video summary, an electronic device, and a storage medium. Background Art

[0002] As a method of condensing videos, video summarization is often used in scenarios such as video preview, video editing, or video clip search. There are two existing ways to generate video summaries. One is to generate video summaries about people or cars for specific categories of targets in specific scenarios, such as traffic scenarios; the other is to learn video summaries that match the scenarios by collecting a large amount of data from various scenarios and annotating them. Existing methods for generating video summaries have disadvantages such as a single scene or a large amount of annotated data. Obviously, the existing methods for generating video summaries will result in different users seeing the same video summaries, which will bring less freshness and a poor experience to users. Summary of the invention

[0003] The embodiments of the present application provide a method, device, electronic device, and storage medium for generating a video summary, so as to enable generating a video summary from different angles according to information of interest.

[0004] In a first aspect, an embodiment of the present application provides a method for generating a video summary, which may include the following steps:

[0005] Determine features of video segments, where the video segments are obtained by segmenting the video;

[0006] Determining features of video image frames in the video clip, the number of video image frames in the video clip being multiple;

[0007] Determine the characteristics of attention information, where the attention information is used to characterize the attention paid to the video in different dimensions;

[0008] A video summary of the video is generated by utilizing features of the video clip, features of the video image frames in the video clip, and features of the attention information.

[0009] In a second aspect, an embodiment of the present application provides a method for generating a video summary, which may include the following steps:

[0010] Sending the received attention information about the video to a generating end of the video summary;

[0011] The video summary of the video generated by the video summary generation end in response to the attention information is received; the video summary of the video is generated by the video summary generation end using the features of the video segments, the features of the video image frames in the video segments, and the features of the attention information; the video segments are obtained by segmenting the video;

[0012] Displays the video summary of the video in the video preview window.

[0013] In a third aspect, an embodiment of the present application provides a device for generating a video summary, which may include:

[0014] A video feature determination module, used to determine the features of a video segment, where the video segment is obtained by segmenting the video;

[0015] An image feature determination module, used to determine features of video image frames in a video clip, wherein the number of video image frames in each video clip is multiple;

[0016] An attention information determination module is used to determine the characteristics of attention information, and the attention information is used to represent the attention paid to the video in different dimensions;

[0017] The video summary generation module is used to generate a video summary of the video by utilizing the features of the video clip, the features of the video image frames in the video clip and the features of the attention information.

[0018] In a fourth aspect, an embodiment of the present application provides a device for generating a video summary, which may include:

[0019] The attention information sending module is used to send the received attention information on the video to the generation end of the video summary;

[0020] The video summary acquisition module is used to receive the video summary of the video generated by the video summary generation end in response to the attention information; the video summary of the video is generated by the video summary generation end using the features of the video clip, the features of the video image frames in the video clip and the features of the attention information; the video clip is obtained by segmenting the video;

[0021] The video summary display module is used to display the video summary of the video in the video preview window.

[0022] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods described above when executing the computer program.

[0023] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the methods described above is implemented.

[0024] Compared with the prior art, this application has the following advantages:

[0025] According to the embodiment of the present application, the video summary is adaptively generated based on the content of different user preferences through the attention information. It is achieved that different users can generate summary videos that meet the user preferences with the same model, thereby meeting the personalized needs of different users. At the same time, in the process of video summary generation, multiple dimensions of information such as the characteristics of video clips, video image frame characteristics, and characteristics of attention information can be referred to. Comprehensive consideration of multi-dimensional information can obtain higher quality video summary results.

[0026] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0028] Figure 1 A schematic diagram of a scenario of a method for generating a video summary provided in this application;

[0029] Figure 2 One of the flowcharts of the method for generating a video summary according to an embodiment of the present application;

[0030] Figure 3 One of the flowcharts of a method for determining a video segment according to an embodiment of the present application;

[0031] Figure 4 This is a second flowchart of a method for determining a video segment according to an embodiment of the present application;

[0032] Figure 5 This is one of the flowcharts of the specific process of generating a video summary according to an embodiment of the present application;

[0033] Figure 6 This is a second flowchart of a specific process of generating a video summary according to an embodiment of the present application;

[0034] Figure 7 This is a flowchart of a specific process of generating a video summary according to an embodiment of the present application;

[0035] Figure 8 This is one of the flow charts of a method for generating attention information according to an embodiment of the present application;

[0036] Fig. 9 This is a second flow chart of a method for generating attention information according to an embodiment of the present application;

[0037] Fig.10 This is the second flowchart of the method for generating a video summary according to an embodiment of the present application;

[0038] Fig.11 This is a third flowchart of a method for generating a video summary according to an embodiment of the present application;

[0039] Fig.12 This is a fourth flowchart of a method for generating a video summary according to an embodiment of the present application;

[0040] Fig.13 This is one of the structural block diagrams of a video summary generation device according to an embodiment of the present application;

[0041] Fig.14 is a second structural block diagram of a video summary generating device according to an embodiment of the present application; and

[0042] Fig.15 A block diagram of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION

[0043] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0044] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0045] First, the terms involved in this application are explained.

[0046] Chinese Multimodal Pre-trained Model (M6, Multi-Modality to Multi-Modality MultitaskMega-Transformer): This model is based on the translation model (Transformer) and is pre-trained through multiple tasks. Pre-training enables the model to have single-modal and multi-modal understanding and generation capabilities. The Chinese Multimodal Pre-trained Model can be applied to a range of downstream applications, for example, it can include description generation for objects, visual question answering, Chinese poetry generation, etc.

[0047] Video feature extraction model (Video Transformer): Based on the self-attention mechanism, it can extract features from the input video clips. The output of the video feature extraction model is the features of the video clips.

[0048] Video Shot-Segmentation algorithm: It relies on the pixel changes, histogram changes, edge changes, etc. between consecutive video frames to segment the video. Each video segment can correspond to a shot switching process in the video.

[0049] Video Scene-Segmentation: As a subtask in video understanding, its main goal is to segment a long video into several video segments based on the scene content. A video segment consists of multiple consecutive shot segments.

[0050] Video Captioning: Video Captioning applies deep learning to the fields of computer vision and natural language processing. Specifically, given a video, the video captioning algorithm outputs text describing the video.

[0051] Figure 1 The figure is a schematic diagram of an exemplary application scenario for implementing the method of the embodiment of the present application. In the process of generating video summaries, different users often have different needs. For example, for the same video, the first user's preference is the actor's clothing combination, and the second user's preference is the food clips in the video. In order to solve this problem of different user preferences, the model is usually adjusted to obtain a new model by re-labeling the training samples and fine-tuning the model parameters, so as to adopt different models to meet the differentiated needs of users. However, the disadvantage of re-labeling the training samples is the increase in the cost of manual labeling and the time cost of model training.

[0052] In this regard, in the current application scenario, the target video for which the video summary is to be generated can be split into multiple video clips according to specified rules. In addition, user preference information can be received. The user's preference information can be received text information, sound information, or video information, etc. The user's preference information is used to characterize the user's preference for the target video in different dimensions. For example, Figure 1The clothing combinations and food clips shown in can represent the preferences of two different users. Multiple video clips of the target video are matched with the preference information. The matching may include extracting features of the video clips, features of the preference information, and comparing the similarities between the features of the video clips and the features of the preference information. Finally, the target video clips related to the user's preferences can be determined. Based on the determined target video clips, after processes such as combining the target video clips, adjusting the image quality, or adjusting the length of the video clips, a video summary that matches the user's preference information can be obtained. For example, in Figure 1 In the example, a video summary related to clothing matching content can be generated for the first user's preference. A video summary related to food can be generated for the second user's preference. This can solve the problem of needing to re-label training data for training fine-tuning due to different scenarios and different user preferences, and greatly reduce the resource consumption caused by multiple trainings for different scenarios and different users. A set of models can be used to achieve adaptive generation of video summaries based on preference information.

[0053] The present application embodiment provides a method for generating a video summary, such as Figure 2 The flowchart of the method for generating a video summary according to an embodiment of the present application may include:

[0054] Step S201: determining features of video segments, where the video segments are obtained by segmenting the video.

[0055] The execution subject of the embodiment of the present application can be the cloud or the client. The video clip can be obtained by cutting the target video to generate the video summary using the specified rules. Among them, the specified rule can be a time rule. For example, each video clip can be 1 minute or 30 seconds, etc. Or, the specified rule can be a shot rule. For example, each shot switching of the target video can correspond to a video clip. Or, the specified rule can also be a scene rule. For example, each scene can correspond to a video clip. The scene can include at least one shot. In the case where the scene includes multiple shots, the contents captured by the multiple shots can be similar. For example, multiple shots are shots taken at the same shooting location, or multiple shots are shots of an actor, etc.

[0056] The features of the video clip can be extracted using a pre-trained video feature extraction model. The video clip is input into the pre-trained video feature extraction model to obtain the features corresponding to the video clip. The features can be represented in the form of a vector.

[0057] Step S202: determining features of video image frames in a video clip, where the number of video image frames in the video clip is multiple.

[0058] Each video segment may include multiple video image frames. The number of video image frames is proportional to the duration of the video segment. The features of the video image frames may include features such as pixel dimensions and features such as text dimensions.

[0059] The features of the pixel dimension can be determined using the previously trained Chinese multimodal pre-trained model. The Chinese multimodal pre-trained model includes an image encoder and a text encoder. The image encoder can be used to directly determine the features of the video image frame from the pixel dimension. The features of the text dimension can be determined using the video description generation algorithm and the Chinese multimodal pre-trained model. The video description generation algorithm can be used to obtain the text description of the specified video image frame in the video clip. The text encoder can be used to obtain the features of the text dimension based on the text description.

[0060] The designated video image frames may be all video image frames of the video clip, or may be image frames obtained at intervals of a certain time or a certain number of frames. For example, a video image frame may be obtained at intervals of 0.1 seconds. For another example, an odd or even number of video image frames may be obtained. For another example, a video image frame may be obtained at intervals of 5 frames, etc.

[0061] Step S203: determining the characteristics of the attention information, where the attention information is used to characterize the attention paid to the video in different dimensions.

[0062] The attention information can be the preference information input by different users or obtained by collecting data from different users, which is used to characterize the attention paid to the video in different dimensions. Taking the football match video between Team A and Team B as an example, User A is a fan of Team A, and User B is a fan of Player C of Team B. Then User A's attention information can be the wonderful performance of Team A or the highlights of Team B's mistakes. User B's attention information can be the wonderful performance of Player C. Taking the video of food exploration as an example, User A's attention information can be the introduction of the types of dishes. For example, the signature dish of the store is BBQ, and how to choose meat and how to marinate meat are the focus of User A. User B's attention information can be the address and environment of the store. For example, it can be the business district where the store is located, whether the store is an Internet celebrity store, whether it is a store on the XX list, etc.

[0063] The features of the information of interest can be determined using a pre-trained Chinese multimodal pre-training model. When the received preference information is voice data, the voice data can be converted into text information first, and then the features of the information of interest can be determined using the text encoder included in the Chinese multimodal pre-training model. When the received preference information is text information, the features of the information of interest can be determined directly using the text encoder included in the Chinese multimodal pre-training model.

[0064] Step S204: Generate a video summary of the video by using the features of the video clip, the features of the video image frames in the video clip, and the features of the attention information.

[0065] Starting from each video image frame and the video clip corresponding to each video image frame, one or more video image frames with a high correlation with the information of interest are determined. The so-called high correlation may be a video image frame with a correlation not lower than a corresponding threshold. A pre-trained scoring model may be used, the input of which is the features of the video clip, the features of the video image frame in the video clip, and the features of the information of interest, and the output is the score of each video image frame.

[0066] If multiple video image frames are screened out according to the scores, it can be further determined whether each video image frame corresponds to the same video clip. If it corresponds to the same video clip, the determined unique video clip can be used to generate a video summary. If it corresponds to multiple video clips, the multiple video clips can be used to generate a video summary. For example, there are 3 video image frames corresponding to the first video clip, and 7 image frames corresponding to the second video clip. Since the number of video image frames in the first video clip is relatively small, the video summary can be generated using only the second video clip. Alternatively, the first video clip and the second video clip can be directly spliced ​​to generate a video summary. Alternatively, the video summary can also be generated using a partially continuous first video clip containing 3 video image frames and a partially continuous second video clip containing 7 image frames. For example, the partially continuous first video clip containing 3 video image frames can be a video clip composed of multiple video image frames adjacent to the front and multiple video image frames adjacent to the back of each of the three video image frames.

[0067] Through the above process, the attention information is used to adaptively generate video summaries based on the content preferred by different users. The same model can meet the needs of different users. At the same time, in the process of matching video clips, the features of video image frames, and the features of attention information can be referred to. Multi-dimensional feature correlation matching can obtain higher quality video summary results.

[0068] like Figure 3 As shown, in a possible implementation, the method of determining the video segment involved in step S201 may include:

[0069] Step S301: according to the shot switching of the video, the video is divided into a plurality of shot segments.

[0070] During the video shooting process, there may be multiple shots. The final video can be obtained by editing, splicing, and other operations on the content shot by multiple shots. For example, in a scene between two actors, the first shot may be a shot of actor A, and the second shot may be a shot of actor B. Multiple shots may switch back and forth in the video. Based on this, the video can be segmented according to the shot switching of the video, with each shot switching as a shot segment segmentation instruction. The video can be segmented into multiple shot segments according to the shot switching situation, which can be achieved using a video shot cutting algorithm.

[0071] Step S302: taking the shot segment as the determined video segment.

[0072] According to the shot switching situation, the video can be divided into a plurality of shot segments. Based on this, each shot segment can be used as a determined video segment.

[0073] like Figure 4 As shown, in a possible implementation, the method of determining the video segment involved in step S201 may also include:

[0074] Step S401: merging the shot segments according to the similarity of the contents of the shot segments to obtain at least one content segment.

[0075] Similar shot clips can be multiple shot clips in the same scene. The same scene may include the same shooting location, the same or similar shooting style, or the same or similar shooting content. Taking a food exploration program as an example, the exterior shot before entering the food store can be classified as the first scene. The shot shot in the dining area of ​​the food store can be classified as the second scene. The shot shot in the back kitchen of the food store can be classified as the third scene. Taking film and television dramas of different themes as an example, the shots of literary dramas can be classified as the first scene, and the shots of martial arts dramas can be classified as the second scene. Each content clip can be composed of at least one shot clip.

[0076] The content segment can be obtained by merging multiple lens segments after the lens segments are determined. In addition, the content segment can also be realized by using a video scene cutting algorithm.

[0077] Step S402: taking the content segment as the determined video segment; or taking the content segment and the shot segment as the determined video segment at the same time.

[0078] In one case, the content segment can be used as the determined video segment. The beneficial effect is that the amount of data can be effectively reduced. In another case, the content segment and the lens segment can be used as the determined video segment at the same time. The beneficial effect is that the features of the video can be annotated from more dimensions.

[0079] like Figure 5 As shown, in a possible implementation, the step S204 involves generating a video summary of the video by utilizing the features of the video clip, the features of the video image frames in the video clip, and the features of the attention information, which may include:

[0080] Step S501: determining at least one key frame in the video image frame by using a first correlation degree between the feature of the video clip and the feature of the attention information and a second correlation degree between the feature of the video image frame and the feature of the attention information.

[0081] The features of the information of interest can be used as reference information. Thus, based on the multi-dimensional features such as the features of the video image frame and the features of the video clip, the features of the information of interest are matched to determine the key frames with a high degree of matching with the information of interest. The so-called high degree of matching with the features of the information of interest may be a video image frame whose matching degree is not less than a corresponding threshold.

[0082] For example, a first correlation degree between the features of the video clip and the features of the information of interest may be determined first. Second, a second correlation degree between the features of the video image frame and the features of the information of interest may be determined. Finally, a score of each video image frame may be determined based on the first correlation degree and the second correlation degree. The video image frame with the highest score may be used as a key frame. Alternatively, a specified number of video image frames may be selected as key frames by sorting from high to low scores.

[0083] Step S502: Generate a video summary of the video according to the video segments corresponding to the key frames.

[0084] The video summary may be obtained by performing corresponding processing on the video segments corresponding to the key frames according to the duration requirement of the video summary. For example, the video segments corresponding to the key frames may be subjected to frame extraction processing, compression processing, etc. according to the duration requirement to obtain a video summary that meets the duration requirement.

[0085] like Figure 6 As shown, in a possible implementation, the step S502 of generating a video summary of the video according to the video segment corresponding to the key frame may include:

[0086] Step S601: Obtain the duration of the video segment corresponding to the key frame.

[0087] First, it can be determined whether the video segments corresponding to each key frame are the same. In the case where there are at least two video segments, the duration of each video segment needs to be obtained respectively.

[0088] Step S602: Filter the video segments corresponding to the key frames according to the determined scores of the key frames and the duration of the video segments corresponding to the key frames to obtain filtering results.

[0089] When there are at least two video clips, the scores of the key frames in each video clip and the duration of the video clip can be used as reference information to filter out at least one video clip. For example, a backpack algorithm can be used to select a video clip with a suitable duration as a screening result based on the scores of the key frames.

[0090] Step S603: Generate a video summary of the video using the screening results.

[0091] After obtaining the screening results, a video summary can be generated by direct splicing. Alternatively, the duration of the video clips in the screening results can be adjusted first, and the video summary can be generated using the video clips after the duration adjustment. The duration adjustment can be performed based on the scores of the key frames. For example, a video clip containing a key frame with a relatively high score can be adjusted relatively less. A video clip containing a key frame with a relatively low score can be adjusted relatively more.

[0092] like Figure 7 As shown, in a possible implementation, when the duration corresponding to the screening result exceeds the corresponding threshold, the step S502 of generating a video summary of the video according to the video segment corresponding to the key frame may include:

[0093] Step S701: filtering the video image frames in the video segment corresponding to the key frame based on the features of the video image frames in the video segment corresponding to the key frame.

[0094] The duration requirement of the video summary can be determined first. If the duration requirement of the video summary is 1 minute, and there are multiple video clips in the screening results, and the total duration of the multiple video clips exceeds 1 minute, the video image frames in the multiple video clips can be filtered. For example, based on the features of the video image frames that have been determined in advance, the video image frames in the multiple video clips can be compared for similarity. When the difference between the video image frames involved in the comparison is not greater than the difference threshold, it can be considered that the video image frames involved in the comparison are similar. Then, the video image frames are filtered according to the comparison results. For example, if there are 10 consecutive video image frames that are similar, the feature mean of the 10 video image frames can be calculated. Then, the 10 video image frames are compared with the feature mean respectively, and the video image frame with the largest difference from the feature mean is retained, or the video image frame with a difference exceeding the corresponding threshold is retained. In addition, the first frame, the last frame and the middle frame can be directly selected as the retained video image frames.

[0095] Step S702: Generate a video summary of the video using the filtered video image frames.

[0096] Since the purpose of filtering is to perform time compression, the number of video image frames retained after filtering meets the requirement. That is, the duration of the video summary obtained by combining the filtered video image frames can meet the duration requirement of the video summary.

[0097] like Figure 8 As shown, in a possible implementation, the method of generating the attention information may include:

[0098] Step S801: determining instruction content and reference content from the acquired video summary generation instruction; the instruction content is generated using at least one of voice data and motion data; and the reference content includes at least one of text, video or image.

[0099] The acquired video summary generation instruction may be information input by the user or may be information collected by the execution subject. Exemplarily, the acquired or collected information may be a (text) image held by the user, or a (handheld) video playback device that is playing a video, etc. The user points to a certain local content in the image and says "generate a video clip similar to this character's state", the user points to or circles a certain text in the image and says "I need such content", points to a certain screen and says "I want a video clip like this", etc. In the above scenarios, the collected user's finger movements, circled movements, and user's voice can all be used as instruction content, and the image, text, or video being played held by the user can be used as reference content.

[0100] Based on this, the acquired video summary generation instruction can be parsed. The user's voice data and action data can be directly determined as the instruction content. The items held by the user, the content pointed to, the circled content, etc. can be determined as the reference content. The above analysis can be determined using a semantic model, and the instruction content and reference content can be determined by semantically identifying the collected content.

[0101] After determining the instruction content and the reference content, the user can also be identified based on the instruction content. For example, the user can be identified through information such as images or sounds, thereby setting an identifier for the user. In the subsequent display of the video summary, the user can be identified based on the video summary display instruction of the user. If the identification result is a user who has output the attention information, the video summary generated by the attention information corresponding to the user can be directly displayed.

[0102] Step S802: Using the instruction content, determine the information of interest in the reference content.

[0103] The content of the instruction can be identified to determine the intent of the instruction. For example, when the content of the instruction is voice data, the intent of the instruction can be determined using voice recognition technology. When the content of the instruction is action data, the action can be identified to determine the intent of the instruction. Among them, the action can include the action pointed to in the aforementioned example, the circled action, etc. In this case, the position pointed by the finger, or the content corresponding to the circled position can be used as the selected reference content. For example, a person in the image, or a building, or a piece of text is circled. The selected person, building, or circled text in the image can be used as attention information. Then, the characteristics of the attention information can be determined by an image encoder, a text encoder, etc.

[0104] like Fig. 9 As shown, in a possible implementation, when the instruction content includes voice data and action data, the step S802 involves using the instruction content to determine the information of interest in the reference content, which may include:

[0105] Step S901: Determine a first time when voice data appears and a second time when motion data appears.

[0106] When the instruction content is determined, the moment when the instruction content appears can also be recorded. By parsing the instruction content, voice data and action data can be distinguished. The parsing principle can be implemented using existing technologies and will not be described in detail. For example, when a user circles a target while speaking to a video being played, the first moment when voice data appears and the second moment when action data appears can be determined accordingly. In the current implementation, the voice data and action data can be content that represents instruction meaning.

[0107] Step S902: using the first time and the second time, the voice data and the motion data are associated to obtain an associated result.

[0108] A time difference threshold can be set in advance. If the time difference between the first moment and the second moment is not greater than the time difference threshold, it can be determined that the voice data and the action data occur simultaneously. That is, simultaneous occurrence can be a situation where the voice data and the action data overlap in the time dimension. Based on this, the voice data and action data that occur simultaneously can be associated to obtain an associated result. The expression form of the association result can be: {starting time t1, ending time t2, voice data, action data}, or, the expression form of the association result can be: {starting time t1, ending time t2, voice data}, {starting time t1, ending time t3, action data}. In the current example, the first time period from time t1 to time t2 overlaps with the second time period from time t1 to time t3. It is not difficult to understand that for instruction contents that do not overlap in time, they can be recorded in sequence based on the chronological order.

[0109] Step S903: using the association result, determine the scope of the reference content; the scope includes at least one of the video duration, the number of image pages, and the effective content of the image.

[0110] As in the above example, if the starting time of the association result is t1, the reference content at t1 can be recorded. If the reference content is a video, the video screen at t1 can be recorded, so that t1 can be used as the starting time of the video. For the continuity of the screen, 1 second before t1 or 2 seconds before t1 can also be used as the starting time of the video.

[0111] In the first time period from t1 to t2, or in the second time period from t1 to t3, the instruction content can be parsed. For example, the voice content includes "I need a composition like this part" or "I want the wonderful performance of this player", etc. The action content includes circling, pointing and other actions in the played video. Based on this, the time tags of the voice data and the action data can be used to load the same time tag for the content displayed in the corresponding time period of the reference content. The content displayed in the corresponding time period can be a video clip, a single or multiple images of corresponding length, etc. Alternatively, the content displayed in the corresponding time period can also be the content pointed to by the voice content and / or the content pointed to by the action content in the time period. If the voice content includes "I need a composition like this part" and "I want the wonderful performance of this player", the composition method of the video or image is the effective content, and the player in the image is the effective content.

[0112] Step S904: using the range of the reference content, determine the information of interest in the reference content.

[0113] In the current implementation, the expression form of the attention information can be text. After determining the scope of the reference content, if the reference content is a video or an image, the text expression of the video or image can be obtained using a video description generation algorithm. If the reference content is text, the text expression can be directly used.

[0114] like Fig.10 As shown, in a possible implementation, when the execution subject is a client, it may also include:

[0115] Step S1001: Associating the video summary with the video.

[0116] According to different attention information, the video summary corresponding to the attention information can be obtained. As mentioned above, the attention information can be clothing matching and food clips. In this regard, for the same target video, a first video summary related to clothing matching and a second video summary related to food can be obtained. For the two video summaries, both can be associated with the target video to indicate that the two video summaries are obtained from the target video.

[0117] Step S1002: upon receiving a video summary display instruction, the video preview window of the video displays the video summary.

[0118] The video summary display instruction may be in the form of voice, action, etc. For example, the voice form may be "play the video summary of a certain video". The action form may be the video of the summary to be played selected by the action. Furthermore, the user who issues the video summary display instruction may be identified. For example, the user who issues the video summary display instruction may be determined by image recognition or sound recognition. Then, it may be determined whether there is already attention information matching the user. If so, the video summary corresponding to the attention information may be displayed in the video preview window of the video. On the contrary, if it does not exist, a video summary with higher popularity may be selected for display.

[0119] The above-mentioned method for generating a video summary can be an application program (APP) local to the user terminal, or a functional module in an APP, or a service provided by the cloud. The user calls the call interface corresponding to the service, uploads the information of interest to the cloud, and receives the result of the cloud feedback, such as the video summary of the video. In addition, the cloud can obtain content related to the video from the video database.

[0120] Exemplarily, several distributed computing nodes may be deployed in the cloud, each of which has computing, storage and other processing resources. In the cloud, multiple computing nodes may be organized to provide one or several services in the card identification method; or, multiple computing nodes may be organized to provide one or several services in the video summary generation method. Exemplarily, the service may include segmenting the video to obtain video segments; determining the characteristics of the video segments; determining the characteristics of the video image frames in the video segments; determining the characteristics of the information of interest; using the characteristics of the video segments, the characteristics of the video image frames in the video segments and the characteristics of the information of interest to generate a video summary of the video, etc. Of course, a computing node may also provide one or more services. The cloud may provide the service by providing a service interface to the outside, and the user may call the service interface to use the corresponding service.

[0121] For the solution provided in the embodiment of the present invention, the cloud can provide a service interface for information identification services, which is called a target service interface. When a user needs to view a video summary, the target service interface is called through the user device to trigger a request to call the target service interface to the cloud, and the request carries the information of interest. The cloud determines a computing node that responds to the request, and uses the processing resources in the computing node to execute the steps provided in the embodiment of the present application.

[0122] The present application embodiment provides a method for generating a video summary, such as Fig.11 The flowchart of the method for generating a video summary according to an embodiment of the present application may include:

[0123] Step S1101: sending the received attention information on the video to the generating end of the video summary.

[0124] The execution subject of the embodiment of the present application may be a client. The attention information may be the preference information for the video issued by the user. The scenarios for issuing the attention information may include video preview scenarios, video editing scenarios, video clip search scenarios, etc. The preference information may be in the form of text information, sound information, or video information, etc. The user's preference information is used to characterize the user's preference for the target video in different dimensions. Exemplarily, the attention information may be a user holding a (text) image, or a (handheld) video playback device that is playing a video, etc. The user points to a local content in the image and says "generate a video clip similar to this character state", the user points to or circles a text in the image and says "I need such content", points to a certain picture and says "I want a video clip like this", etc.

[0125] Step S1102: receiving a video summary of a video generated by a video summary generation end in response to the attention information; the video summary of the video is generated by the video summary generation end using features of video segments, features of video image frames in the video segments, and features of the attention information; the video segments are obtained by segmenting the video.

[0126] The video summary generation end is used to generate a video summary of the video based on the attention information of the video. In the process of generating the video summary, the features of the video segment and the features of the video image frames in the video segment can also be referred to at the same time.

[0127] The video clip can be obtained by segmenting the video for which the video summary is to be generated using a specified rule. The features of the video clip can be extracted using a pre-trained video feature extraction model. The video clip is input into the pre-trained video feature extraction model to obtain the features corresponding to the video clip. The features can be represented in the form of a vector.

[0128] Each video segment may include multiple video image frames. The number of video image frames is proportional to the duration of the video segment. The features of the video image frames may include features such as pixel dimensions and features such as text dimensions.

[0129] The features of the pixel dimension can be determined using the previously trained Chinese multimodal pre-trained model. The Chinese multimodal pre-trained model includes an image encoder and a text encoder. The image encoder can be used to directly determine the features of the video image frame from the pixel dimension. The features of the text dimension can be determined using the video description generation algorithm and the Chinese multimodal pre-trained model. The video description generation algorithm can be used to obtain the text description of the specified video image frame in the video clip. The text encoder can be used to obtain the features of the text dimension based on the text description.

[0130] The features of the attention information can be determined using a pre-trained Chinese multimodal pre-training model. When the received attention information is voice data, the voice data can be converted into text information first, and then the features of the attention information can be determined using the text encoder contained in the Chinese multimodal pre-training model. When the received attention information is text information, the features of the attention information can be determined directly using the text encoder contained in the Chinese multimodal pre-training model. When the received attention information is action information, the action information can be converted into text information based on action recognition technology, and then the features of the attention information can be determined using the text encoder contained in the Chinese multimodal pre-training model.

[0131] A pre-trained scoring model is used, the input of which is the features of the video clip, the features of the video image frame in the video clip, and the features of the attention information, and the output is the score of each video image frame. According to the score of the video image frame and the duration of the video image frame video clip, the target video clip can be determined using the backpack algorithm. That is, the video summary of the video generated by the video summary generation end in response to the attention information. The video summary generation end sends the video summary to the client. The functions and specific workflow of the video summary generation end can be found in the corresponding description of the above method, and have corresponding beneficial effects, which will not be repeated here.

[0132] Step S1103: Display the video summary of the video in the video preview window.

[0133] After receiving the video summary of the video sent by the generating end of the video summary in response to the attention information, the client can display the video summary of the video in the video preview window.

[0134] By focusing on information, we can adaptively generate video summaries based on the content preferences of different users, and meet the needs of different users with the same model.

[0135] The present application embodiment provides a method for generating a video summary, such as Fig.12 The flowchart of the method for generating a video summary according to an embodiment of the present application may include:

[0136] Use the image feature extraction model to determine the pixel dimension features of the image frames in the target video Wherein j can be used to represent the sequence number of the image frame.

[0137] Using the video description generation algorithm, the text representation T of the image frame in the target video is determined.

[0138] Using the text feature extraction model, we can get the features of the text representation T of the image frame. Pixel dimension features and characters representing the characteristics of T can be used together as image features.

[0139] The attention information can be the user's preference information. After the attention information is converted into text, the features of the attention information are extracted using the text feature extraction model.

[0140] Using the video shot cutting algorithm, the target video can be divided into multiple shot segments, and the duration of the shot segment can be expressed as T shot Using the video feature extraction model, we can get the features of the shot clips Where i can be used to represent the sequence number of the shot clip.

[0141] Using the video scene segmentation algorithm, the target video can be segmented into multiple scene segments, each of which can include at least one shot segment. The duration of the scene segment can be expressed as T sence Using the video feature extraction model, we can get the features of the scene clips Wherein q can be used to represent the sequence number of the scene segment.

[0142] The pixel dimension feature Characteristic of text representation T Focus on the characteristics of information Features of a shot Features of scene clips By inputting a pre-trained scoring model, the score Sj of the j-th image frame can be obtained. The pre-trained scoring model can be a neural network model based on the translation model (Transformer) structure. The neural network model may include an input layer, an encoding layer, and an output layer. The input layer may be various features. The encoding layer may use an encoder of the translation model structure. The translation model structure has a self-attention mechanism, which can directly model the mutual influence between any two features. The output layer can generate a score for each image frame based on the result of the encoding layer. By sorting based on the scores, the score Sj of the j-th image frame can be obtained. For a frame of image with a higher score or a specified number of frames of image, it can be used to generate a video summary.

[0143] Using the knapsack algorithm, based on the score Sj of the image frame and the duration T of the shot segment shot , we can get the footage that is suitable as the video summary. Finally, based on the duration requirement of the video summary, we can generate the video summary.

[0144] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a device for generating a video summary. Fig.13 The structure block diagram of a device for generating a video summary according to an embodiment of the present application is shown. The device for generating a video summary may include:

[0145] The video feature determination module 1301 is used to determine the features of video segments, where the video segments are obtained by segmenting the video.

[0146] The image feature determination module 1302 is used to determine features of video image frames in a video segment, and there are multiple video image frames in each video segment.

[0147] The attention information determination module 1303 is used to determine the characteristics of the attention information, where the attention information is used to represent the attention paid to the video in different dimensions.

[0148] The video summary generating module 1304 is used to generate a video summary of the video by using the features of the video clip, the features of the video image frames in the video clip and the features of the attention information.

[0149] In a possible implementation, the video feature determination module 1301 may include:

[0150] The sub-slicing module is used to divide the video into multiple shot segments according to the shot switching of the video.

[0151] The video segment determination submodule is used to use the shot segment as the determined video segment.

[0152] In a possible implementation, the video feature determination module 1301 may further include:

[0153] A content segment determination submodule, used to merge the shot segments according to the similarity of the contents of the shot segments to obtain at least one content segment;

[0154] The video segment determination submodule is further configured to use the content segment as the determined video segment; or,

[0155] The content segment and the lens segment are simultaneously regarded as the determined video segment.

[0156] In a possible implementation, the video summary generating module 1304 may include:

[0157] A key frame determination submodule, configured to determine at least one key frame in the video image frame by using a first correlation degree between a feature of the video segment and a feature of the information of interest, and a second correlation degree between a feature of the video image frame and a feature of the information of interest;

[0158] The video summary generation execution submodule is used to generate a video summary of the video according to the video clips corresponding to the key frames.

[0159] In a possible implementation, the video summary generation execution submodule may include:

[0160] A duration acquisition unit, used to acquire the duration of the video segment corresponding to the key frame;

[0161] A screening result determination unit, configured to screen the video segments corresponding to the key frames according to the determined scores of the key frames and the duration of the video segments corresponding to the key frames, to obtain screening results;

[0162] The video summary generating unit is used to generate a video summary of the video using the screening result.

[0163] In a possible implementation, when the duration of the video segment corresponding to the key frame exceeds the corresponding threshold, the video summary generation execution submodule may include:

[0164] A filtering unit, configured to filter the video image frames in the video segment corresponding to the key frame based on the features of the video image frames in the video segment corresponding to the key frame;

[0165] The video summary generating unit is used to generate a video summary of the video by using the filtered video image frames.

[0166] In a possible implementation, the attention information determination module 1303 may include:

[0167] The content acquisition submodule is used to determine the instruction content and the reference content from the acquired video summary generation instruction; the instruction content is generated by using at least one of voice data and motion data; the reference content includes at least one of text, video or image;

[0168] The focus information determination execution submodule is used to determine the focus information in the reference content using the instruction content.

[0169] In a possible implementation, when the instruction content includes voice data and action data, focusing on the information to determine the execution submodule may include:

[0170] a time determination unit, for determining a first time when the voice data appears and a second time when the motion data appears;

[0171] An associating unit, used to associate the voice data with the motion data using the first time and the second time to obtain an association result;

[0172] A range determination unit, used to determine the range of the reference content using the association result; the range includes at least one of the video duration, the number of image pages, and the effective content of the image;

[0173] The attention information determination unit is configured to determine the attention information in the reference content by using the range of the reference content.

[0174] In a possible implementation, a display module is further included, and the display module may include:

[0175] an associating subunit, for associating the video summary with the video;

[0176] The display execution subunit is used to display the video summary in the video preview window of the video when receiving the video summary display instruction.

[0177] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a device for generating a video summary. Fig.14 The structure block diagram of a device for generating a video summary according to an embodiment of the present application is shown. The device for generating a video summary may include:

[0178] The attention information sending module 1401 is used to send the received attention information on the video to the generating end of the video summary;

[0179] The video summary acquisition module 1402 is used to receive the video summary of the video generated by the video summary generation end in response to the attention information; the video summary of the video is generated by the video summary generation end using the features of the video segment, the features of the video image frames in the video segment and the features of the attention information; the video segment is obtained by segmenting the video;

[0180] The video summary display module 1403 is used to display the video summary of the video in the video preview window.

[0181] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0182] Fig.15 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Fig.15 As shown, the electronic device includes: a memory 1510 and a processor 1520. The memory 1510 stores a computer program that can be run on the processor 1520. When the processor 1520 executes the computer program, the method in the above embodiment is implemented. The number of the memory 1510 and the processor 1520 can be one or more.

[0183] The electronic device also includes:

[0184] The communication interface 1530 is used to communicate with external devices and perform data exchange transmission.

[0185] If the memory 1510, the processor 1520 and the communication interface 1530 are implemented independently, the memory 1510, the processor 1520 and the communication interface 1530 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0186] Optionally, in a specific implementation, if the memory 1510, the processor 1520 and the communication interface 1530 are integrated on a chip, the memory 1510, the processor 1520 and the communication interface 1530 can communicate with each other through an internal interface.

[0187] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0188] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0189] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0190] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0191] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0192] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0193] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0194] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0195] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0196] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0197] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0198] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0199] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a video summary, characterized in that: include: Determining features of a video segment, wherein the video segment is obtained by segmenting the video; Determining features of video image frames in the video clip, wherein the number of video image frames in the video clip is multiple; Determine features of attention information, where the attention information is used to characterize attention to the video in different dimensions; A video summary of the video is generated by utilizing the features of the video clip, the features of the video image frames in the video clip, and the features of the information of interest; wherein, the step of generating the video summary of the video by utilizing the features of the video clip, the features of the video image frames, and the features of the information of interest comprises: determining at least one key frame in the video image frames by utilizing a first degree of association between the features of the video clip and the features of the information of interest, and a second degree of association between the features of the video image frames and the features of the information of interest; and generating the video summary of the video according to the video clip corresponding to the key frame.

2. The method according to claim 1, characterized in that The method for determining the video segment includes: According to the shot switching of the video, the video is divided into a plurality of shot segments; The shot segment is used as the determined video segment.

3. The method according to claim 2, characterized in that The method for determining the video segment also includes: Merging the shot fragments according to similarities in the contents of the shot fragments to obtain at least one content fragment; Using the content segment as the determined video segment; or, The content segment and the shot segment are simultaneously used as determined video segments.

4. The method according to claim 1, characterized in that Generating a video summary of the video according to the video segment corresponding to the key frame includes: Obtaining the duration of the video segment corresponding to the key frame; Filtering the video segments corresponding to the key frames according to the determined scores of the key frames and the duration of the video segments corresponding to the key frames to obtain filtering results; A video summary of the video is generated using the screening results.

5. The method according to claim 1 or 4, characterized in that: When the duration of the video segment corresponding to the key frame exceeds a corresponding threshold, generating a video summary of the video according to the video segment corresponding to the key frame includes: Based on the features of the video image frames in the video segment corresponding to the key frame, filtering the video image frames in the video segment corresponding to the key frame; A video summary of the video is generated using the filtered video image frames.

6. The method according to claim 1, characterized in that The method for generating the attention information includes: Determining instruction content and reference content from the acquired video summary generation instruction; the instruction content is generated using at least one of voice data and motion data; the reference content includes at least one of text, video or image; The instruction content is used to identify the information of interest in the reference content.

7. The method according to claim 6, characterized in that In the case where the instruction content includes voice data and action data, the using the instruction content to determine the information of interest in the reference content includes: Determine a first time when the voice data appears and a second time when the motion data appears; Associating the voice data with the action data using the first time and the second time to obtain an association result; Determine the range of the reference content by using the association result; the range includes at least one of the video duration, the number of image pages, and the effective content of the image; Using the range of the reference content, information of interest is identified in the reference content.

8. The method according to claim 1, characterized in that Also includes: Associating the video summary with the video; When a video summary display instruction is received, the video summary is displayed in a video preview window of the video.

9. A method for generating a video summary, characterized in that: include: Sending the received attention information about the video to a generating end of the video summary; The video summary of the video generated by the generating end of the video summary in response to the attention information is received; the video summary of the video is generated by the generating end of the video summary using the features of the video segments, the features of the video image frames in the video segments and the features of the attention information; the video segments are obtained by segmenting the video; wherein the generating end of the video summary generates the video summary of the video using the features of the video segments, the features of the video image frames and the features of the attention information, including: the generating end of the video summary determines at least one key frame in the video image frame using a first correlation degree between the features of the video segments and the features of the attention information, and a second correlation degree between the features of the video image frame and the features of the attention information; the generating end of the video summary generates the video summary of the video according to the video segments corresponding to the key frames; A video summary of the video is displayed in the video preview window.

10. A device for generating a video summary, characterized in that: include: A video feature determination module, used to determine features of a video segment, wherein the video segment is obtained by segmenting the video; An image feature determination module, used to determine features of video image frames in the video clips, wherein the number of video image frames in each of the video clips is multiple; An attention information determination module, used to determine features of attention information, where the attention information is used to characterize attention to the video in different dimensions; A video summary generation module is used to generate a video summary of the video by using the features of the video clip, the features of the video image frames in the video clip and the features of the information of interest; wherein, the generating the video summary of the video by using the features of the video clip, the features of the video image frames and the features of the information of interest comprises: determining at least one key frame in the video image frame by using a first correlation degree between the features of the video clip and the features of the information of interest, and a second correlation degree between the features of the video image frame and the features of the information of interest; and generating the video summary of the video according to the video clip corresponding to the key frame.

11. A device for generating a video summary, characterized in that: include: The attention information sending module is used to send the received attention information on the video to the generation end of the video summary; The video summary acquisition module is used to receive the video summary of the video generated by the video summary generation end in response to the attention information; the video summary of the video is generated by the video summary generation end using the characteristics of the video segment, the characteristics of the video image frame in the video segment and the characteristics of the attention information; the video segment is obtained by segmenting the video; wherein the video summary generation end uses the characteristics of the video segment, the characteristics of the video image frame and the characteristics of the attention information to generate the video summary of the video, including: the video summary generation end uses the first correlation degree between the characteristics of the video segment and the characteristics of the attention information, and the second correlation degree between the characteristics of the video image frame and the characteristics of the attention information to determine at least one key frame in the video image frame; the video summary generation end generates the video summary of the video according to the video segment corresponding to the key frame; The video summary display module is used to display the video summary of the video in the video preview window.

12. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

13. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images