Artificial intelligence-based video highlight clip editing method, system and storage medium

Abstract

The present application provides a method, system and storage medium for editing video highlights based on artificial intelligence. The method obtains video data (using H264 encoding mode, VBR bit rate control mode, and a set frame rate); performs feature extraction on the video data to determine the file comprehensive bit rate, video average bit rate, audio average bit rate, parameter ratio of the video bit rate at each moment to the video bit rate at the previous moment, parameter ratio of the audio bit rate at each moment to the audio bit rate at the previous moment, parameter ratio of the video bit rate at each moment to the video bit rate at the next moment, and parameter ratio of the audio bit rate at each moment to the audio bit rate at the next moment, and forms input features based on the obtained information; inputs the input features into a preset video highlight segment recognition model to determine highlight segments and non-highlight segments in the video data. In this way, the extraction of large quantities of video highlight segments can be completed intelligently and efficiently.

Description

Technical Field

[0001] The present application relates to the field of video editing technology, and in particular to an artificial intelligence-based video highlight clip editing method, system, and storage medium. Background Art

[0002] With the development of mobile Internet, video has become the most mainstream media communication method, and the emergence of short videos in recent years has become one of the main ways for people to watch videos. On major video websites, new media industries, large-screen TVs and other terminals, attracting traffic through short videos has also become an important means for various media industries to promote user growth. In view of the need for traffic, the content quality of short video production also plays a decisive role in the effectiveness of user traffic. How to use technical means to quickly realize the recognition of highlights in videos has become an important technical means actively explored by the video industry. Therefore, the video industry has an increasing demand for efficient and intelligent video highlight editing solutions. As for the analysis of highlights in videos, there are generally several types of processing methods:

[0003] 1) The common approach to intelligently identifying video highlights is video image analysis. First, the video is split into multiple segments of a set length. The image information in the video is analyzed, and based on a preset aesthetic evaluation (i.e., a dataset constructed from standard data on dimensions such as composition, lighting, and color, which is an important reference standard for achieving highlight clip capabilities), the segments are identified as highlights. This is a solution centered on video image analysis.

[0004] However, video image analysis solutions, which are usually based on video image analysis solutions, require video segmentation, and the more segments there are, the more precise the analysis of the highlights will be. Therefore, in order to accurately identify the highlights of a video, it is often necessary to segment the video in large quantities, and then perform image analysis on the captured video segments. This also often requires judging the degree of the highlights of the images based on preset video image aesthetic evaluations. For example, images with people and animals appearing, or images with people laughing expressions are preferentially identified as highlights and captured in the highlights. This solution requires a large number of video segments, which consumes a lot of server performance, and requires pre-setting the evaluation criteria for highlights based on user needs, resulting in more restrictions on the capture of highlights and a relatively simple form.

[0005] 2) Another common approach to intelligently identifying highlights of video clips is to extract key frames from the video. First, the video stream is divided into multiple segments to obtain multiple video clips. Key frames are extracted from each segment. For the extracted key frames, the image feature vectors of each key frame are identified. Based on the feature vectors, a deep learning video model is used to screen out target video clips that match a certain classification in the deep learning model. This is a solution centered on key frame analysis of video images.

[0006] However, similar to the first solution, the video keyframe analysis solution also typically requires video segmentation. After segmentation, a large number of keyframes are captured from the segmented videos. The greater the number of segments, the more keyframes are captured, and the more refined the analysis of highlights. Therefore, to accurately identify video highlights, it is often necessary to perform a large number of video segmentation and keyframe extraction. The feature vectors of each keyframe image are then analyzed, and the feature vectors are used to determine whether the image meets the required highlight criteria. This solution also requires a large number of video segments and keyframe extractions, which is also very consuming to server performance.

[0007] Therefore, how to achieve efficient editing of wonderful videos is one of the problems that need to be solved in this field. Summary of the Invention

[0008] The purpose of the embodiments of the present application is to provide a method, system and storage medium for editing video highlights based on artificial intelligence. Based on video bit rate parameters, artificial intelligence technology is used to construct a video highlight recognition model to intelligently and efficiently complete the extraction of large quantities of video highlights.

[0009] In order to achieve the above objectives, the embodiments of the present application are implemented in the following manner:

[0010] In a first aspect, an embodiment of the present application provides a method for editing video highlights based on artificial intelligence, comprising: acquiring video data, wherein the video data adopts an H264 encoding mode, a VBR bitrate control mode, and a set frame rate, and the video data includes a video timestamp; performing feature extraction on the video data to determine the file comprehensive bitrate, video average bitrate, audio average bitrate, a parameter ratio of the video bitrate at each moment to the video bitrate at the previous moment, a parameter ratio of the audio bitrate at each moment to the audio bitrate at the previous moment, a parameter ratio of the video bitrate at each moment to the video bitrate at the next moment, and a parameter ratio of the audio bitrate at each moment to the audio bitrate at the next moment, and forming input features accordingly, wherein the file comprehensive bitrate includes the video bitrate and audio bitrate of the video data at each moment; inputting the input features of the video data into a preset video highlight recognition model to determine highlight clips and non-highlight clips in the video data, and editing the highlight clips.

[0011] In combination with the first aspect, in a first possible implementation method of the first aspect, a video highlight segment recognition model is constructed as follows: obtaining at least one type of video material set, wherein each video material in each type of video material set adopts H264 encoding mode, VBR bit rate control mode, and a set frame rate, and each video material at least contains marked highlight segment timestamps and non-highlight segment timestamps, and the duration of each highlight segment is not less than the set duration; for each video material in each type of video material set, based on the highlight segment timestamps and non-highlight segment timestamps of the video material, the Class I segment and Class II segment of this video material are cut out, wherein each Class I segment contains the time period where the highlight segment is located, the pre-highlight segment time period, The post-highlight segment period, each Class II segment includes the non-highlight segment period, the non-highlight segment pre-period, and the non-highlight segment post-period; for each Class I segment of each video material: feature extraction is performed on the Class I segment to determine the file comprehensive bit rate, video average bit rate, audio average bit rate, and the parameter ratio of the video bit rate at the start time of the highlight segment period to the video bit rate at the adjacent time in the pre-highlight segment period, the parameter ratio of the audio bit rate at the start time of the highlight segment period to the audio bit rate at the adjacent time in the pre-highlight segment period, the parameter ratio of the video bit rate at the end time of the highlight segment period to the video bit rate at the adjacent time in the post ... The parameter ratio of the audio bit rate at the end moment of the highlight segment to the audio bit rate at the adjacent moment in the post-highlight segment period, and based on this, the input features corresponding to the highlight segments in the Class I segments are formed, wherein the file comprehensive bit rate includes the video bit rate and audio bit rate at each moment in the period where the highlight segment is located; for each Class II segment of each video material: feature extraction is performed on the Class II segments to determine the file comprehensive bit rate, video average bit rate, audio average bit rate of the period where the non-highlight segment is located, as well as the parameter ratio of the video bit rate at the start moment of the period where the non-highlight segment is located to the video bit rate at the adjacent moment in the pre-highlight segment period, and the parameter ratio of the audio bit rate at the start moment of the period where the non-highlight segment is located to the audio bit rate at the adjacent moment in the pre-highlight segment period The method comprises the following steps: first, determining the parameter ratio of the video bit rate at the end of the time period where the non-highlight segment is located to the video bit rate at the adjacent time period after the non-highlight segment, and second, determining the parameter ratio of the audio bit rate at the end of the time period where the non-highlight segment is located to the audio bit rate at the adjacent time period after the non-highlight segment, and forming the input features corresponding to the non-highlight segments in the Class II segments accordingly, wherein the comprehensive file bit rate includes the video bit rate and audio bit rate at each moment in the time period where the non-highlight segment is located; based on the input features corresponding to the highlight segments in each Class I segment of each video material in each type of video material set and the input features corresponding to the non-highlight segments in each Class II segment, the deep learning data model is trained and tested to finally obtain a trained video highlight segment recognition model.

[0012] In combination with the first possible implementation method of the first aspect, in the second possible implementation method of the first aspect, when the type of video material set is one, the deep learning data model is trained and tested based on the input features corresponding to the wonderful clips in each Class I clip of each video material in each type of video material set and the input features corresponding to the non-highlight clips in each Class II clip, and finally a trained video highlight clip recognition model is obtained, including: dividing the input features corresponding to the wonderful clips in each Class I clip of each video material in the video material set and the input features corresponding to the non-highlight clips in each Class II clip into a training set and a test set, wherein each input feature corresponds to a label, and the label reveals whether the input feature corresponds to a wonderful clip or a non-highlight clip; building an AIGC deep learning data model; using the training set to train the AIGC deep learning data model, and using the test set to test the trained AIGC deep learning data model, and finally obtaining a trained video highlight clip recognition model.

[0013] In combination with the first possible implementation method of the first aspect, in the third possible implementation method of the first aspect, when there are multiple types of video material sets, the deep learning data model is trained and tested based on the input features corresponding to the wonderful segments in each Class I segment of each video material in each type of video material set and the input features corresponding to the non-highlight segments in each Class II segment, and finally a trained video highlight segment recognition model is obtained, including: for each type of video material set: the input features corresponding to the wonderful segments in each Class I segment of each video material in the video material set and the input features corresponding to the non-highlight segments in each Class II segment are divided into a training set and a test set, wherein each input feature corresponds to a label, and the label reveals whether the input feature corresponds to a wonderful segment or a non-highlight segment; building multiple AIGC deep learning sub-models, wherein the number of AIGC deep learning sub-models is consistent with the number of types of video material sets; using the training set and test set corresponding to each type of video material set to train and test one AIGC deep learning sub-model respectively, to obtain a trained AIGC deep learning sub-model; all AIGC deep learning sub-models are "parallel" merged into a large model as a video highlight segment recognition model.

[0014] In combination with the third possible implementation method of the first aspect, in a fourth possible implementation method of the first aspect, an AIGC deep learning sub-model is trained and tested using the training set and test set corresponding to each type of video material set to obtain a trained AIGC deep learning sub-model, including: for each AIGC deep learning sub-model: training the AIGC deep learning sub-model using the training set corresponding to the corresponding type of video material set; testing the AIGC deep learning sub-model using the test set corresponding to the corresponding type of video material set; after manually calibrating the test results, adding them to the training set for secondary training until the accuracy requirements are met, thereby completing the training of the AIGC deep learning sub-model.

[0015] In combination with the fourth possible implementation method of the first aspect, in the fifth possible implementation method of the first aspect, the AIGC deep learning sub-model is trained using the training set corresponding to the corresponding type of video material set, including: grouping the input features in the training set corresponding to the corresponding type of video material set according to the label to obtain highlight training data and non-highlight training data; using the highlight training data and non-highlight training data to train the AIGC deep learning sub-model using the "independent first and then cross" training mode, wherein the "independent first and then cross" training mode means first training the AIGC deep learning sub-model with the highlight training data, then training the AIGC deep learning sub-model with the non-highlight training data, and finally training the AIGC deep learning sub-model with the mixed highlight training data and non-highlight training data.

[0016] In combination with the fourth possible implementation of the first aspect, in a sixth possible implementation of the first aspect, the AIGC deep learning sub-model is an LSTM model, a BERT model, or a GRU model.

[0017] In a second aspect, an embodiment of the present application provides an artificial intelligence-based video highlight clipping system, comprising: a data acquisition unit for acquiring video data, wherein the video data adopts an H264 encoding mode, a VBR bit rate control mode, and a set frame rate, and the video data includes a video timestamp; a feature extraction unit for performing feature extraction on the video data, determining the file comprehensive bit rate, the video average bit rate, the audio average bit rate, the parameter ratio of the video bit rate at each moment to the video bit rate at the previous moment, the parameter ratio of the audio bit rate at each moment to the audio bit rate at the previous moment, the parameter ratio of the video bit rate at each moment to the video bit rate at the next moment, and the parameter ratio of the audio bit rate at each moment to the audio bit rate at the next moment, and forming input features accordingly, wherein the file comprehensive bit rate includes the video bit rate and audio bit rate of the video data at each moment; a clip clipping unit for inputting the input features of the video data into a preset video highlight clip recognition model, determining highlight clips and non-highlight clips in the video data, and clipping the highlight clips.

[0018] In a third aspect, an embodiment of the present application provides a storage medium, which is arranged in an electronic device, and the storage medium includes a stored program, wherein when the program is running, the electronic device where the storage medium is located is controlled to execute the artificial intelligence-based video highlight clip editing method described in any one of the first aspect or possible implementation methods of the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, and when the program instructions are loaded and executed by the processor, the method for editing video highlights based on artificial intelligence is implemented according to the first aspect or any one of the possible implementation methods of the first aspect.

[0020] Beneficial effects:

[0021] 1. This solution obtains video data (using H264 encoding mode, VBR bitrate control mode, and a set frame rate, including video timestamps); performs feature extraction on the video data to determine the video data's comprehensive file bitrate (including the video and audio bitrates at each moment), the average video bitrate, the average audio bitrate, the parameter ratio of the video bitrate at each moment to the video bitrate at the previous moment, the parameter ratio of the audio bitrate at each moment to the audio bitrate at the previous moment, the parameter ratio of the video bitrate at each moment to the video bitrate at the next moment, and the parameter ratio of the audio bitrate at each moment to the audio bitrate at the next moment, and forms input features based on these features; inputs the input features of the video data into a preset video highlight segment recognition model to identify highlight and non-highlight segments in the video data and edit the highlight segments. By using the relevant parameter changes of the video bitrate to analyze video highlight segments and establish a video highlight segment recognition model, it can quickly and accurately analyze highlight segments in the video intelligently, consume fewer server resources, and be more lightweight and convenient to deploy. Compared with the manual analysis method used by video editors to extract highlights, this solution can greatly reduce the workload of video editors and intelligently and efficiently complete the extraction of large quantities of video highlights.

[0022] 2. This solution is based on the analysis of video bit rate parameters to achieve intelligent recognition and editing of video highlights. Compared with image analysis technology, the edited highlights are richer in scene form, avoiding the shortcoming of the single scene form of the existing method. Highlights can be identified in various image scenes with large differences. At the same time, combined with audio bit rate parameters, the reference latitude is more comprehensive than that of image recognition technology, and the edited highlights are richer in scenes. In addition, this solution can perform differentiated highlight recognition for different types of videos (through different sub-models to identify highlights for different types of videos), and has stronger practicality. Based on bit rate parameter analysis, the video bit rate parameter is read and calculated. Compared with the technology using image recognition, the performance requirements for the device GPU are greatly reduced, saving equipment costs.

[0023] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of a method for editing video highlights based on artificial intelligence is provided in an embodiment of the present application.

[0026] Figure 2 This is a structural block diagram of an electronic device provided in an embodiment of the present application.

[0027] Figure 3 A schematic diagram of an artificial intelligence-based video highlight clip editing system provided in an embodiment of the present application.

[0028] Icons: 10-artificial intelligence-based video highlight clip editing system; 11-data acquisition unit; 12-feature extraction unit; 13-clip editing unit; 20-electronic device; 21-memory; 22-communication module; 23-bus; 24-processor. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0030] See also Figure 1 , Figure 1 The flowchart of a method for editing video highlights based on artificial intelligence provided by an embodiment of the present application is as follows: The method for editing video highlights based on artificial intelligence may include step S10, step S20 and step S30.

[0031] The video highlights clipping method based on artificial intelligence can be applied to electronic devices 20, see Figure 2 , Figure 2 This is a structural block diagram of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may be a computer or a server, which is not limited here.

[0032] For example, electronic device 20 may include a communication module 22 connected to the outside world via a network, one or more processors 24 for executing program instructions, a bus 23, and various forms of memory 21, such as a disk, ROM, or RAM, or any combination thereof. Memory 21, communication module 22, and processor 24 may be connected via bus 23.

[0033] Exemplarily, programs are stored in the memory 21. The processor 24 can call and run these programs from the memory 21, thereby realizing the video highlight clip editing method based on artificial intelligence by running the programs.

[0034] To facilitate understanding of this solution, before introducing the video highlight clip editing method based on artificial intelligence, the video highlight clip recognition model is first introduced here.

[0035] First, the electronic device 20 can obtain at least one type of video material set, wherein each video material in each type of video material set adopts H264 encoding mode, VBR bit rate control mode, and a set frame rate, and each video material contains at least marked highlight segment timestamps and non-highlight segment timestamps, and the duration of each highlight segment is not less than the set duration.

[0036] For example, each original video material in the video material set may have inconsistencies in mode, frame rate, etc. Therefore, each original video material in the video material set can be batch preprocessed, and the encoding mode of each video material can be set to H264, the video bit rate control mode to VBR, the audio bit rate control mode to VBR, and the frame rate can be unified (for example, 25 frames per second), thereby obtaining video materials that meet the requirements to ensure the consistency and accuracy of subsequent analysis.

[0037] Each video material contains at least a timestamp of the highlights (used to identify highlights) and a timestamp of the non-highlights (used to identify non-highlights), and the duration of each highlight is not less than a set duration (this set duration can be set to different values ​​based on different types of video materials, and the specific value is set as needed). As for the number of highlights and non-highlights marked in each video material, there are generally multiple highlights and non-highlights.

[0038] Then, for each video clip in each type of video clip set:

[0039] The electronic device 20 can cut out Class I segments and Class II segments of the video material based on the timestamps of the highlight segments and the timestamps of the non-highlight segments of the video material, wherein each Class I segment includes the time period where the highlight segment is located, the time period before the highlight segment, and the time period after the highlight segment, and each Class II segment includes the time period where the non-highlight segment is located, the time period before the non-highlight segment, and the time period after the non-highlight segment.

[0040] For example, the highlight timestamp can be used to crop the time period of the highlight. The start time of the highlight period can be pushed forward a certain length (e.g., 3 seconds) to crop the pre-highlight period. The end time of the highlight period can be pushed back a certain length (e.g., 3 seconds) to crop the post-highlight period. A clip consisting of the highlight period, pre-highlight period, and post-highlight period is a Category I clip.

[0041] Similarly, the timestamps of non-highlight segments can be used to trim the time period of the non-highlight segments. By moving the start time of the non-highlight segment forward a certain length (e.g., 3 seconds), the pre-non-highlight segment period can be trimmed. By moving the end time of the non-highlight segment backward a certain length (e.g., 3 seconds), the post-non-highlight segment period can be trimmed. A segment consisting of the non-highlight segment period, the pre-non-highlight segment period, and the post-non-highlight segment period is a Category II segment.

[0042] In this way, the type I segments and type II segments corresponding to each video material in each type of video material set can be determined.

[0043] Then, for each Category I segment of each video material:

[0044] The electronic device 20 can perform feature extraction on the Class I clips, and determine the comprehensive bit rate of the file in the time period where the highlight clip is located (including the video bit rate and audio bit rate at each moment in the time period where the highlight clip is located), the average video bit rate, the average audio bit rate, and the parameter ratio of the video bit rate at the start moment in the time period where the highlight clip is located to the video bit rate at the adjacent moment in the preceding time period of the highlight clip (that is, the parameter ratio of the video bit rate at the start moment in the time period where the highlight clip is located to the previous moment, where the previous moment is located in the preceding time period of the highlight clip in the Class I clip), the parameter ratio of the audio bit rate at the start moment in the time period where the highlight clip is located to the audio bit rate at the adjacent moment in the preceding time period of the highlight clip (that is, ... start moment in the time period where the highlight clip is located to the audio bit rate at the adjacent moment in the preceding time period of the highlight clip The parameter ratio of the audio bit rate at the previous moment, the previous moment is located in the pre-highlight period of the I-type segment), the parameter ratio of the video bit rate at the end moment of the highlight period to the video bit rate at the adjacent moment in the post-highlight period (that is, the parameter ratio of the video bit rate at the end moment of the highlight period to the moment after it, the moment after it is located in the post-highlight period of the I-type segment), the parameter ratio of the audio bit rate at the end moment of the highlight period to the audio bit rate at the adjacent moment in the post-highlight period (that is, the parameter ratio of the audio bit rate at the end moment of the highlight period to the moment after it, the moment after it is located in the post-highlight period of the I-type segment). Then, the electronic device 20 can form the input features corresponding to the highlight segments in the I-type segment based on this.

[0045] Of course, in order to enrich the input features corresponding to the highlight clips, more specific parameters can also be extracted, such as the parameter ratio of the video bit rate and the parameter ratio of the audio bit rate at each moment in the pre-time period of the highlight clip and its previous moment, the parameter ratio of the video bit rate and the parameter ratio of the audio bit rate at each moment in the post-time period of the highlight clip and its subsequent moment, etc.

[0046] And, for each Category II segment of each video material:

[0047] The electronic device 20 can perform feature extraction on the Class II segments, and determine the comprehensive bit rate of the file in the time period where the non-highlight segments are located (including the video bit rate and audio bit rate at each moment in the time period where the non-highlight segments are located), the average video bit rate, the average audio bit rate, and the parameter ratio of the video bit rate at the starting moment in the time period where the non-highlight segments are located to the video bit rate at the adjacent moment in the preceding time period of the non-highlight segments (i.e., the parameter ratio of the video bit rate at the starting moment in the time period where the non-highlight segments are located to the previous moment, where the previous moment is in the preceding time period of the non-highlight segments in the Class II segments), the parameter ratio of the audio bit rate at the starting moment in the time period where the non-highlight segments are located to the audio bit rate at the adjacent moment in the preceding time period of the non-highlight segments (i.e., ... starting moment in the time period where the non-highlight segments are located to the audio bit rate at the adjacent moment in the preceding time period of the non-highlight segments The parameter ratio of the audio bit rate at the previous moment, the previous moment is located in the non-highlight segment pre-time period of the Class II segment), the parameter ratio of the video bit rate at the end moment of the non-highlight segment time period to the video bit rate at the adjacent moment in the non-highlight segment post-time period (that is, the parameter ratio of the video bit rate at the start moment of the non-highlight segment time period to the moment after it, the next moment is located in the non-highlight segment post-time period of the Class II segment), the parameter ratio of the audio bit rate at the end moment of the non-highlight segment time period to the audio bit rate at the adjacent moment in the non-highlight segment post-time period (that is, the parameter ratio of the audio bit rate at the start moment of the non-highlight segment time period to the moment after it, the next moment is located in the non-highlight segment post-time period of the Class II segment). Then, the electronic device 20 can form the input features corresponding to the non-highlight segments in the Class II segment based on this.

[0048] Similarly, in order to enrich the input features corresponding to non-highlight clips, more specific parameters can also be extracted, such as the parameter ratio of the video bit rate and the parameter ratio of the audio bit rate at each moment in the period before the non-highlight clip and its previous moment, the parameter ratio of the video bit rate and the parameter ratio of the audio bit rate at each moment in the period after the non-highlight clip and its subsequent moment, etc.

[0049] After determining the input features corresponding to the highlight segments in each Category I segment of each video material in each type of video material set and the input features corresponding to the non-highlight segments in each Category II segment, the electronic device 20 can train and test the deep learning data model accordingly, and finally obtain a trained video highlight segment recognition model.

[0050] Exemplarily, when the type of the video material set is one, the electronic device 20 can divide the input features corresponding to the wonderful segments in each Class I segment of each video material in the video material set and the input features corresponding to the non-wonderful segments in each Class II segment into a training set and a test set (for example, divided in a ratio of 7:3), wherein each input feature corresponds to a label, and the label reveals whether the input feature corresponds to a wonderful segment or a non-wonderful segment.

[0051] Next, we build an AIGC (Artificial Intelligence GPT-3.5Container), a tool and platform for deep learning. It is based on the GPT-3.5 model and provides a series of functions and interfaces that can be used for various deep learning tasks. A deep learning data model (such as an LSTM model, BERT model, or GRU model) is then trained using the training set and tested using the test set, ultimately yielding a trained video highlight recognition model.

[0052] The training strategy can be to first use the training samples of the highlight clips in the training set (i.e., the input features corresponding to Class I clips) to train the AIGC deep learning data model. After the recognition of highlight clips reaches the set accuracy, the training samples of the non-highlight clips in the training set (i.e., the input features corresponding to Class II clips) are used to train the AIGC deep learning data model. After the recognition of non-highlight clips reaches the set accuracy, the training samples in the training set (i.e., the input features corresponding to Class I clips and the input features corresponding to Class II clips) are used to cross-train the AIGC deep learning data model, so that the AIGC deep learning data model can achieve the set accuracy in recognizing highlight clips and non-highlight clips, thus completing the training of the AIGC deep learning data model. The trained AIGC deep learning data model can then be tested using the test set, and the test results are manually verified and calibrated. They are then added as training samples to the training set for secondary training until the training requirements are met. In this way, the training and testing of the AIGC deep learning data model are completed, and a trained video highlight clip recognition model is obtained.

[0053] For example, when there are multiple types of video material sets, for each type of video material set:

[0054] The electronic device 20 can divide the input features corresponding to the wonderful segments in each Class I segment of each video material in the video material set and the input features corresponding to the non-wonderful segments in each Class II segment into a training set and a test set (for example, divided in a ratio of 7:3), wherein each input feature corresponds to a label, and the label reveals whether the input feature corresponds to a wonderful segment or a non-wonderful segment.

[0055] Then, build multiple AIGC deep learning sub-models (such as LSTM model, BERT model or GRU model, etc.), where the number of AIGC deep learning sub-models is consistent with the number of types of video material sets.

[0056] Then, an AIGC deep learning sub-model is trained and tested using the training set and test set corresponding to each type of video material set to obtain a trained AIGC deep learning sub-model.

[0057] For each AIGC deep learning sub-model:

[0058] The electronic device 20 can use the training set corresponding to the corresponding type of video material set to train the AIGC deep learning sub-model. The training strategy can be to group the input features in the training set corresponding to the corresponding type of video material set according to the label to obtain highlight training data and non-highlight training data. The highlight training data and non-highlight training data are trained on the AIGC deep learning sub-model using the "first independent and then cross" training mode, wherein the "first independent and then cross" training mode means first training the AIGC deep learning sub-model with the highlight training data, then training the AIGC deep learning sub-model with the non-highlight training data, and finally training the AIGC deep learning sub-model with the mixed highlight training data and non-highlight training data. For the specific process, please refer to the training strategy introduced above for the case where the type of video material set is one, which will not be described here.

[0059] Afterwards, the AIGC deep learning sub-model can be tested using the test set corresponding to the corresponding type of video material set, and then the test results can be manually calibrated (for example, checking and correcting the timestamps of the highlights and non-highlights), and then added to the training set for secondary training until the accuracy requirements are met and the training of the AIGC deep learning sub-model is completed.

[0060] After training all AIGC deep learning sub-models, they can be combined into a large model in parallel as a video highlight recognition model.

[0061] After obtaining the video highlight recognition model, the video highlight recognition model can be built into the electronic device 20 to run the video highlight editing method based on artificial intelligence.

[0062] In order to achieve intelligent editing of video highlights, the electronic device 20 may execute step S10.

[0063] Step S10: Acquire video data, wherein the video data adopts H264 encoding mode, VBR bit rate control mode, and a set frame rate, and the video data includes a video timestamp.

[0064] In this embodiment, the electronic device 20 can obtain video data (or a collection of multiple video data). Since the video data may have inconsistent modes, frame rates, etc., the video data can be pre-processed first, and the video data encoding mode is set to H.264, the video bit rate control mode is set to VBR, the audio frequency bit rate control mode is set to VBR, and the frame rate is unified (for example, 25 frames per second), thereby obtaining video data that meets the requirements. In addition, each video data has a corresponding type label, such as common media types such as TV series, movies, variety shows, animation, sports events, documentaries, etc. The type label of the video data is one of the type labels of the AIGC deep learning sub-model in the video highlight segment recognition model.

[0065] After obtaining the video data, the electronic device 20 may execute step S20.

[0066] Step S20: Perform feature extraction on the video data to determine the file comprehensive bit rate, video average bit rate, audio average bit rate, parameter ratio of the video bit rate at each moment to the video bit rate at the previous moment, parameter ratio of the audio bit rate at each moment to the audio bit rate at the previous moment, parameter ratio of the video bit rate at each moment to the video bit rate at the next moment, parameter ratio of the audio bit rate at each moment to the audio bit rate at the next moment, and form input features accordingly, wherein the file comprehensive bit rate includes the video bit rate and audio bit rate of the video data at each moment.

[0067] In this embodiment, the electronic device 20 can perform feature extraction on the video data, determine the comprehensive file bit rate of the video data (including the video bit rate and audio bit rate of the video data at each moment), the average video bit rate, the average audio bit rate, the parameter ratio of the video bit rate at each moment to the video bit rate at the previous moment, the parameter ratio of the audio bit rate at each moment to the audio bit rate at the previous moment, the parameter ratio of the video bit rate at each moment to the video bit rate at the next moment, and the parameter ratio of the audio bit rate at each moment to the audio bit rate at the next moment, and form input features based on this.

[0068] It should be noted that, in the case of batch processing, feature extraction can be performed on each video data to obtain corresponding input features (each input feature has a corresponding type label) so that batch processing can be performed subsequently using the video highlight recognition model.

[0069] After obtaining the input features corresponding to the video data, the electronic device 20 may execute step S30.

[0070] Step S30: Inputting the input features of the video data into a preset video highlight segment recognition model, determining highlight segments and non-highlight segments in the video data, and editing the highlight segments.

[0071] In this embodiment, the electronic device 20 can input the input features of the video data into a preset video highlight recognition model, so that the video highlight recognition model can identify highlight segments and non-highlight segments based on the input features, thereby realizing the editing of highlight segments.

[0072] For batch processing, the type label of the input feature (revealing the type of video data, such as TV series, movies, variety shows, animation, sports events, documentaries) can be used to enable the video highlight recognition model to use the corresponding type of sub-model to identify highlights and non-highlights of this type of input features (determine the timestamps of highlights and non-highlights), so as to edit the corresponding highlights and non-highlights based on the timestamps of highlights and non-highlights.

[0073] It should be noted that the use of a video highlight segment recognition model in this solution to determine highlight segments and non-highlight segments in video data is an exemplary solution. Under the concept of this solution, it is not only possible to identify highlight segments and non-highlight segments, but also to achieve more specific classification, for example, to identify segments of multiple different highlight levels (such as highlight segments, normal segments, non-highlight segments; or, S-level highlight segments, A-level highlight segments, B-level highlight segments, C-level highlight segments, D-level highlight segments, etc.). Of course, during the model training phase, it is necessary to label the corresponding segment classifications in order to train the model, thereby achieving the recognition and editing of segments of different highlight levels. Therefore, the recognition of highlight segments and non-highlight segments introduced in this embodiment cannot be regarded as the solution being able to only recognize highlight segments and non-highlight segments. As long as the recognition and editing of highlight video segments are carried out in accordance with the concept of this solution, it should also fall within the scope of protection of this solution.

[0074] See also Figure 3 Based on the same inventive concept, the embodiment of the present application further provides an artificial intelligence-based video highlight clip editing system 10, comprising:

[0075] The data acquisition unit 11 is used to acquire video data, wherein the video data adopts H264 encoding mode, VBR bit rate control mode, and a set frame rate, and the video data includes a video timestamp.

[0076] The feature extraction unit 12 is used to extract features from the video data, determine the file comprehensive bit rate, video average bit rate, audio average bit rate, parameter ratio of the video bit rate at each moment to the video bit rate at the previous moment, parameter ratio of the audio bit rate at each moment to the audio bit rate at the previous moment, parameter ratio of the video bit rate at each moment to the video bit rate at the next moment, parameter ratio of the audio bit rate at each moment to the audio bit rate at the next moment, and form input features accordingly, wherein the file comprehensive bit rate includes the video bit rate and audio bit rate of the video data at each moment.

[0077] The segment editing unit 13 is configured to input the input features of the video data into a preset video highlight segment recognition model, determine highlight segments and non-highlight segments in the video data, and edit the highlight segments.

[0078] In addition, an embodiment of the present application also provides a storage medium, which is set in an electronic device 20, and the storage medium includes a stored program, wherein when the program is running, the electronic device 20 where the storage medium is located is controlled to execute an artificial intelligence-based video highlight clip editing method.

[0079] In summary, the embodiments of the present application provide a method, system and storage medium for editing video highlights based on artificial intelligence, which obtains video data (using H264 encoding mode, VBR bit rate control mode, and a set frame rate, including a video timestamp); performs feature extraction on the video data to determine the file comprehensive bit rate of the video data (including the video bit rate and audio bit rate of the video data at each moment), the average video bit rate, the average audio bit rate, the parameter ratio of the video bit rate at each moment to the video bit rate of the previous moment, the parameter ratio of the audio bit rate at each moment to the audio bit rate of the previous moment, the parameter ratio of the video bit rate at each moment to the video bit rate of the next moment, and the parameter ratio of the audio bit rate at each moment to the audio bit rate of the next moment, and form input features accordingly; input the input features of the video data into a preset video highlight recognition model to determine the highlight and non-highlight segments in the video data, and edit the highlight segments. By analyzing video highlights based on changes in video bitrate parameters and establishing a video highlight recognition model, this solution can quickly and accurately analyze video highlights, consume fewer server resources, and is more lightweight and convenient to deploy. Compared to manual analysis by video editors to extract highlights, this solution significantly reduces the workload for video editors and intelligently and efficiently extracts high-volume video highlights.

[0080] This solution is based on the analysis of video bitrate parameters to achieve intelligent recognition and editing of video highlights. Compared with image analysis technology, the edited highlights are richer in scene form, avoiding the shortcoming of the single scene form of the existing method. Highlights can be identified in various image scenes with large differences. At the same time, combined with audio bitrate parameters, the reference latitude is more comprehensive than that of image recognition technology, and the edited highlights are richer in scenes. In addition, this solution can perform differentiated highlight recognition for different types of videos (through different sub-models to identify highlights for different types of videos), and has stronger practicality. Based on bitrate parameter analysis, the video bitrate parameter is read and calculated. Compared with the technology using image recognition, the performance requirements for the device GPU are greatly reduced, saving equipment costs.

[0081] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0082] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A video highlight clip editing method based on artificial intelligence, characterized in that: include: Obtain video data, wherein the video data adopts H264 encoding mode, VBR bit rate control mode, and a set frame rate, and the video data includes a video timestamp; Performing feature extraction on the video data to determine a file comprehensive bit rate, an average video bit rate, an average audio bit rate, a parameter ratio of the video bit rate at each moment to the video bit rate at the previous moment, a parameter ratio of the audio bit rate at each moment to the audio bit rate at the previous moment, a parameter ratio of the video bit rate at each moment to the video bit rate at the next moment, and a parameter ratio of the audio bit rate at each moment to the audio bit rate at the next moment, and forming input features based on the features, wherein the file comprehensive bit rate includes the video bit rate and audio bit rate of the video data at each moment; Inputting the input features of the video data into a preset video highlight segment recognition model, determining highlight segments and non-highlight segments in the video data, and editing the highlight segments; The video highlights recognition model is constructed as follows: Obtain at least one type of video material set, wherein each video material in each type of video material set uses the H264 encoding mode, the VBR bitrate control mode, and a set frame rate, and each video material at least includes a timestamp of a highlight segment and a timestamp of a non-highlight segment, and the duration of each highlight segment is not less than the set duration; For each video material in each type of video material set, based on the timestamp of the highlight segment and the timestamp of the non-highlight segment of the video material, the type I segment and the type II segment of the video material are cut out, wherein each type I segment includes the time period of the highlight segment, the time period before the highlight segment, and the time period after the highlight segment; each type II segment includes the time period of the non-highlight segment, the time period before the non-highlight segment, and the time period after the non-highlight segment; For each Category I segment of each video material: feature extraction is performed on the Category I segment to determine the comprehensive file bit rate, average video bit rate, average audio bit rate of the time period where the highlight segment is located, as well as the parameter ratio of the video bit rate at the start moment of the time period where the highlight segment is located to the video bit rate at the adjacent moment in the time period before the highlight segment, the parameter ratio of the audio bit rate at the start moment of the time period where the highlight segment is located to the audio bit rate at the adjacent moment in the time period before the highlight segment, the parameter ratio of the video bit rate at the end moment of the time period where the highlight segment is located to the video bit rate at the adjacent moment in the time period after the highlight segment, and the parameter ratio of the audio bit rate at the end moment of the time period where the highlight segment is located to the audio bit rate at the adjacent moment in the time period after the highlight segment, and form the input features corresponding to the highlight segment in the Category I segment accordingly, wherein the comprehensive file bit rate includes the video bit rate and audio bit rate at each moment in the time period where the highlight segment is located; For each Category II segment of each video material: feature extraction is performed on the Category II segment to determine the file comprehensive bit rate, video average bit rate, and audio average bit rate of the time period where the non-highlight segment is located, as well as the parameter ratio of the video bit rate at the start moment of the time period where the non-highlight segment is located to the video bit rate at the adjacent time moments in the preceding time period of the non-highlight segment, the parameter ratio of the audio bit rate at the start moment of the time period where the non-highlight segment is located to the audio bit rate at the adjacent time moments in the preceding time period of the non-highlight segment, the parameter ratio of the video bit rate at the end moment of the time period where the non-highlight segment is located to the video bit rate at the adjacent time moments in the following time period of the non-highlight segment, and the parameter ratio of the audio bit rate at the end moment of the time period where the non-highlight segment is located to the audio bit rate at the adjacent time moments in the following time period of the non-highlight segment, and thereby form input features corresponding to the non-highlight segment in the Category II segment, wherein the file comprehensive bit rate includes the video bit rate and audio bit rate at each moment in the time period where the non-highlight segment is located; Based on the input features corresponding to the highlight clips in each Category I segment of each video material in each type of video material set and the input features corresponding to the non-highlight clips in each Category II segment, the deep learning data model is trained and tested, and finally a trained video highlight clip recognition model is obtained.

2. The method for editing video highlights based on artificial intelligence according to claim 1, characterized in that: When the video material set is of one type, the deep learning data model is trained and tested based on the input features corresponding to the highlight segments in each Category I segment and the input features corresponding to the non-highlight segments in each Category II segment of each video material in each type of video material set. Finally, a trained video highlight segment recognition model is obtained, including: The input features corresponding to the highlight segments in each Category I segment and the input features corresponding to the non-highlight segments in each Category II segment of each video material in the video material set are divided into a training set and a test set, wherein each input feature has a corresponding label, and the label reveals whether the input feature corresponds to a highlight segment or a non-highlight segment; Build AIGC deep learning data model; The AIGC deep learning data model is trained using the training set, and the trained AIGC deep learning data model is tested using the test set, ultimately obtaining a trained video highlight segment recognition model.

3. The method for editing video highlights based on artificial intelligence according to claim 1, characterized in that: When there are multiple types of video material sets, the deep learning data model is trained and tested based on the input features corresponding to the highlight segments in each Category I segment and the input features corresponding to the non-highlight segments in each Category II segment of each video material in each type of video material set. The trained video highlight segment recognition model is finally obtained, including: For each type of video material set: the input features corresponding to the highlight segments in each Category I segment and the input features corresponding to the non-highlight segments in each Category II segment of each video material in the video material set are divided into a training set and a test set, wherein each input feature has a corresponding label, and the label indicates whether the input feature corresponds to a highlight segment or a non-highlight segment; Build multiple AIGC deep learning sub-models, where the number of AIGC deep learning sub-models is consistent with the number of types of video material sets; Use the training set and test set corresponding to each type of video material set to train and test an AIGC deep learning sub-model respectively to obtain a trained AIGC deep learning sub-model; All AIGC deep learning sub-models are "parallelized" into a large model as a video highlight segment recognition model.

4. The method for editing video highlights based on artificial intelligence according to claim 3, characterized in that: The training set and test set corresponding to each type of video material set are used to train and test an AIGC deep learning sub-model respectively, and the trained AIGC deep learning sub-model is obtained, including: For each AIGC deep learning sub-model: Use the training set corresponding to the corresponding type of video material set to train the AIGC deep learning sub-model; The AIGC deep learning sub-model is tested using the test set corresponding to the corresponding type of video material set; After manual calibration of the test results, they are added to the training set for secondary training until the accuracy requirements are met, completing the training of the AIGC deep learning sub-model.

5. The method for editing video highlights based on artificial intelligence according to claim 4, characterized in that: The AIGC deep learning sub-model is trained using the training set corresponding to the corresponding type of video material set, including: Group the input features in the training set corresponding to the corresponding type of video material set according to the labels to obtain the highlight clip training data and the non-highlight clip training data; The AIGC deep learning sub-model is trained using the "independent first, then cross" training mode with the highlight training data and the non-highlight training data. The "independent first, then cross" training mode means that the AIGC deep learning sub-model is first trained with the highlight training data, then trained with the non-highlight training data, and finally trained with the mixed highlight training data and non-highlight training data.

6. The method for editing video highlights based on artificial intelligence according to claim 4, characterized in that: The AIGC deep learning sub-model is an LSTM model, a BERT model, or a GRU model.

7. An artificial intelligence-based video highlight clip editing system, characterized in that: include: A data acquisition unit is used to acquire video data, wherein the video data adopts H264 encoding mode, VBR bit rate control mode, and a set frame rate, and the video data includes a video timestamp; a feature extraction unit, configured to perform feature extraction on the video data, determine a file comprehensive bit rate, an average video bit rate, an average audio bit rate, a parameter ratio of the video bit rate at each moment to the video bit rate at the previous moment, a parameter ratio of the audio bit rate at each moment to the audio bit rate at the previous moment, a parameter ratio of the video bit rate at each moment to the video bit rate at the next moment, and a parameter ratio of the audio bit rate at each moment to the audio bit rate at the next moment, and form input features accordingly, wherein the file comprehensive bit rate includes the video bit rate and audio bit rate of the video data at each moment; a clip editing unit, configured to input the input features of the video data into a preset video highlight segment recognition model, determine highlight segments and non-highlight segments in the video data, and edit the highlight segments; The video highlight segment recognition model is constructed as follows: obtaining at least one type of video material set, wherein each video material in each type of video material set adopts H264 encoding mode, VBR bit rate control mode, and has a set frame rate, and each video material contains at least marked highlight segment timestamps and non-highlight segment timestamps, and the duration of each highlight segment is not less than the set duration; for each video material in each type of video material set, based on the highlight segment timestamps and non-highlight segment timestamps of the video material, the Class I segment and Class II segment of the video material are cut out, wherein each Class I segment contains the time period of the highlight segment, the pre-highlight segment time period, and the post-highlight segment time period, and each Class II segment contains Including the time period where non-highlight clips are located, the time period before non-highlight clips, and the time period after non-highlight clips; for each I-type clip of each video material: feature extraction is performed on the I-type clip to determine the file comprehensive bit rate, video average bit rate, audio average bit rate of the time period where the highlight clip is located, as well as the parameter ratio of the video bit rate at the starting moment of the time period where the highlight clip is located to the video bit rate at the adjacent time in the time period before the highlight clip, the parameter ratio of the audio bit rate at the starting moment of the time period where the highlight clip is located to the audio bit rate at the adjacent time in the time period before the highlight clip, the parameter ratio of the video bit rate at the end moment of the time period where the highlight clip is located to the video bit rate at the adjacent time in the time period after the highlight clip, the parameter ratio of the audio bit rate at the end moment of the time period where the highlight clip is located to the audio bit rate at the The parameter ratio of the audio bit rate of the adjacent moments in the post-highlight period, and based on this, form the input features corresponding to the highlight segments in the Class I segments, wherein the file comprehensive bit rate includes the video bit rate and audio bit rate at each moment in the period where the highlight segments are located; for each Class II segment of each video material: feature extraction is performed on the Class II segments to determine the file comprehensive bit rate, video average bit rate, audio average bit rate of the period where the non-highlight segments are located, as well as the parameter ratio of the video bit rate at the starting moment of the period where the non-highlight segments are located to the video bit rate at the adjacent moments in the pre-highlight period, the parameter ratio of the audio bit rate at the starting moment of the period where the non-highlight segments are located to the audio bit rate at the adjacent moments in the pre-highlight period, and the parameter ratio of the non-highlight segments to the audio bit rate at the starting moment of the period where the non-highlight segments are located, The parameter ratio of the video bit rate at the end moment of the time period where the clip is located to the video bit rate at the adjacent moment in the time period after the non-highlight clip, and the parameter ratio of the audio bit rate at the end moment of the time period where the non-highlight clip is located to the audio bit rate at the adjacent moment in the time period after the non-highlight clip, are used to form the input features corresponding to the non-highlight clips in the Class II clips, wherein the comprehensive bit rate of the file includes the video bit rate and audio bit rate at each moment in the time period where the non-highlight clip is located; based on the input features corresponding to the highlight clips in each Class I clip of each video material in each type of video material set and the input features corresponding to the non-highlight clips in each Class II clip, the deep learning data model is trained and tested to finally obtain a trained video highlight clip recognition model.

8. A storage medium, characterized in that: The storage medium is set in the electronic device, and the storage medium includes a stored program, wherein when the program is running, the electronic device where the storage medium is located is controlled to execute the video highlight clip editing method based on artificial intelligence as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, the method for editing video highlights based on artificial intelligence according to any one of claims 1 to 6 is implemented.

Images