Method and system for extracting highlights in long video

Abstract

The invention belongs to the technical field of video image processing, and particularly relates to a method for extracting highlights in a long video, which comprises the following steps of: inputting a plurality of pictures containing a target object and a long video needing to be edited, and carrying out feature extraction on the pictures containing the target object; taking the target object as a detection main body, performing target object video clip identification in a long video needing to be edited, and confirming a starting moment and an ending moment of a wonderful video clip of the target object; based on the starting time and the ending time of the wonderful video clip of the target object, clipping the long video to obtain a target object slice video; and merging the plurality of slice videos of the target object into one video according to a time sequence, and outputting the wonderful fragments of the target object.

Description

Technical Field

[0001] The present invention belongs to the technical field of video image processing, and in particular relates to a method and system for extracting wonderful clips from a long video. Background Art

[0002] With the development of the Internet, building structures tend to be intelligent. By installing a large number of monitoring devices for monitoring and management, the monitoring devices will generate a large amount of video data. The processing of this data will not only consume a lot of server resources, but also make it difficult to mine and extract the clips that users are interested in from the lengthy videos.

[0003] The prior art often identifies the event subject, which cannot generate a collection of highlights for individual users and cannot meet the personalized needs of users to watch their own video highlights. At present, manual processing is often used to capture highlights, but it is time-consuming and labor-intensive to retrieve and extract clips from a large number of videos. Summary of the invention

[0004] The present invention provides a method and system for extracting wonderful clips from a long video. The method extracts features of a target object and uses it as a detection subject to identify video clips in a long video to be edited, obtains sliced ​​videos to be cropped, and synthesizes and outputs wonderful clips of the target object in chronological order. An intelligent method is used to directly output exclusive wonderful clips for the target object, thereby increasing user experience, being convenient, fast, and accurate, and ensuring the quality and continuity of the output video.

[0005] A method for extracting wonderful clips from a long video, comprising:

[0006] Input multiple pictures containing target objects and long videos to be edited, and perform feature extraction on the pictures containing target objects;

[0007] Taking the target object as the detection subject, identify the target object video clips in the long video that needs to be edited, and confirm the start and end time of the target object's wonderful video clips;

[0008] Based on the start and end times of the wonderful video clips of the target object, the long video is clipped to obtain the sliced ​​video of the target object;

[0009] Combine multiple sliced ​​videos of the target object in chronological order to form a video and output the highlight clips of the target object.

[0010] By extracting the features of the target object and taking it as the detection subject, video clips are identified in the long video that needs to be edited, the sliced ​​videos that need to be cropped are obtained, and the highlight clips of the target object are synthesized and output in chronological order; an intelligent method is used to directly output exclusive highlight clips for the target object, which increases the user experience, is convenient, fast, accurate, and ensures the quality and continuity of the output video.

[0011] Furthermore, taking the target object as the detection subject, identifying the wonderful video clips of the target object in the long video to be edited, and confirming the start time and end time of the wonderful video clips of the target object, including:

[0012] Taking the target object as the detection subject, the target object features are identified in the long video to be edited, and the start and end times of the target object video clip are confirmed;

[0013] Based on the target object video clip, the target object wonderful content is identified, and the video clip where the target object wonderful content is located is determined as the target object wonderful video clip.

[0014] By first performing feature recognition on the target object, confirming the video segment where the target object is located, and then performing wonderful content recognition on the video segment, it is convenient to determine the wonderful video content of the target object.

[0015] Furthermore, the identifying of the target object video segment in the long video to be edited includes using an opencv recognition processing model to identify the target object video segment in the long video to be edited, specifically:

[0016] Based on the timeline of the long video, the long video to be edited is divided into blocks in an orderly manner to obtain multiple block videos;

[0017] Based on multiple block videos, binary division is used to extract video frames as key frames in a timely manner;

[0018] The target object is taken as the detection subject, and the target object features are recognized on the key frames.

[0019] By using the binary method to extract video frames as key frames in real time, the key frames where the target object is located can be quickly located, thereby facilitating feature recognition of the key frames to obtain video clips of the target object.

[0020] Furthermore, the identifying of the target object video segment in the long video to be edited also includes using a deep learning recognition processing model to identify the target object video segment in the long video to be edited, specifically including:

[0021] Based on the video frames of the long video, obtain the motion vector of each video frame;

[0022] Based on the acquired motion vector differences under adjacent video frames, a preset abnormality threshold is compared to determine the motion abnormality area; based on the motion abnormality area, a noise reduction process is performed to obtain a corrected video frame;

[0023] Based on the corrected video frame, the target object is taken as the detection subject to perform target object feature recognition.

[0024] By identifying motion anomalies in the video, obtaining the motion abnormality area, and then performing noise reduction processing, the accuracy of video feature recognition is improved.

[0025] Furthermore, the identifying of the target object video segment in the long video to be edited also includes using a third-party platform recognition processing model to identify the target object video segment in the long video to be edited, specifically:

[0026] Load the long video to be recognized and decode it to obtain multiple pictures of continuous video frames;

[0027] Use AI vision algorithms to identify target objects in pictures of multiple consecutive video frames;

[0028] The target tracking algorithm is used to continuously track the target object, and the adjacent video frame images containing the target object are associated to obtain the video clip containing the target object, and the target object features are recognized.

[0029] By introducing a third-party platform to perform target object feature recognition, video clips of the target object can be directly obtained.

[0030] Further, the determining of the start time and the end time of the target object video clip includes:

[0031] When a video frame containing a target object is detected and identified, a search is performed backwards based on the time point of the current video frame until the target object is no longer detected and identified, and the time point at which the target object's video position is no longer detected and identified is recorded as the start time;

[0032] When a video frame containing a target object is detected and identified, the search continues forward based on the time point of the current video frame until the target object is no longer detected and identified, and the time point when the target object's video position is no longer detected and identified is recorded as the end time.

[0033] Further, the method of identifying the wonderful content of the target object based on the target object video clip and determining the video clip containing the wonderful content of the target object as the wonderful video clip of the target object includes:

[0034] Based on the target object video clip, when the calculated motion difference between the current video frame and the adjacent video frame exceeds a preset motion threshold, the current video frame is defined as the first wonderful content;

[0035] Extract the audio track of the target object video clip, perform audio feature analysis, and define the current video frame as the second exciting content when the calculated audio feature difference between the current video frame and the adjacent video frame exceeds a preset audio feature threshold, or the audio feature value of the current video frame is within a preset audio feature range;

[0036] When the current video frame belongs to both the first wonderful content and the second wonderful content, the target object video segment where the current video frame is located is determined as the target object wonderful video segment.

[0037] By comparing the amount of motion between video frames with a preset motion threshold, and the audio features with a preset audio feature threshold, it is helpful to determine the content of the highlight segment of the target object.

[0038] A system for extracting a wonderful segment from a long video, comprising:

[0039] An input and preprocessing module, which is used to input multiple pictures containing target objects and long videos that need to be edited, and perform feature extraction on the pictures containing target objects;

[0040] The recognition processing module is used to identify the target object video clips in the long video to be edited, and confirm the start and end time of the target object's wonderful video clips;

[0041] A cropping module is used to crop the long video based on the start time and end time of the target object's wonderful video clip to obtain the target object slice video;

[0042] The integrated output module is used to merge multiple sliced ​​videos of the target object into one video in chronological order and output the highlight clips of the target object.

[0043] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0044] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0045] The beneficial effects of the present invention are:

[0046] The present invention extracts features of the target object and uses it as the detection subject to identify video clips in a long video that needs to be edited. It uses an intelligent method and multiple identification processing modules to identify the target object, thereby improving the adaptability of the solution. It not only quickly and accurately locates the key frame through the appearance of the target object to shorten the processing time, but also combines the multi-dimensional analysis method of motion and audio features to accurately capture the wonderful moments in the video, and synthesizes the sliced ​​videos that need to be cropped into the wonderful clips of the target object in chronological order, thereby ensuring the quality and continuity of the output video. By removing noise from the video, the accuracy of target object recognition is improved, thereby facilitating efficient and accurate recognition and extraction of wonderful video clips of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the present invention;

[0048] Figure 2 A flowchart for extracting the highlights of the target object using the openvc recognition processing model;

[0049] Figure 3 A flowchart for extracting the highlight clips of the target object using a deep learning recognition processing model;

[0050] Figure 4 A flowchart for extracting the highlight clips of the target object using a third-party platform recognition processing model;

[0051] Figure 5 It is a structural schematic diagram of the system in the present invention;

[0052] Figure 6 A schematic diagram of a computer device. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0055] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples, and a person of ordinary skill in the art can understand the specific meanings of the above terms in this application under specific circumstances.

[0056] Example 1

[0057] Figure 1 The method is a method for extracting wonderful clips from a long video. The method extracts the features of the target object and uses it as the detection subject. The video clips are identified in the long video to be edited, and the sliced ​​videos to be cropped are obtained. The wonderful clips of the target object are synthesized and output in chronological order. The method directly outputs the exclusive wonderful clips for the target object in an intelligent way, which increases the user experience. It is convenient, fast and accurate, and ensures the quality and continuity of the output video. The method includes the following steps:

[0058] S1: Input multiple pictures containing target objects and long videos to be edited, and perform feature extraction on the pictures containing target objects;

[0059] In this embodiment, before extracting features from a picture containing a target object, the target object in the picture is also identified, and the identification is performed manually by an operator.

[0060] S2: Taking the target object as the detection subject, identify the target object video clip in the long video to be edited, and confirm the start and end time of the target object's wonderful video clip;

[0061] S21: Taking the target object as the detection subject, the target object feature recognition is performed in the long video that needs to be edited;

[0062] S211: Taking the target object as the detection subject, the opencv recognition processing model is used to identify the target object video clip in the long video that needs to be edited;

[0063] S2111: Sequentially divide the long video to be edited into blocks based on the time axis of the long video to obtain multiple block videos;

[0064] S2112: Based on the multiple block videos, extract video frames as key frames using binary division method in a timely manner;

[0065] In this embodiment, a binary search method is adopted to extract a video frame as a key frame every 3 seconds.

[0066] S2113: Taking the target object as the detection subject, the target object features are identified on the key frame.

[0067] S212: Taking the target object as the detection subject, a deep learning recognition processing model is used to identify the target object video clip in the long video to be edited;

[0068] In this embodiment, before using the deep learning model for denoising, the deep learning model needs to be pre-trained to improve the accuracy of the model output.

[0069] S2121: Based on the video frames of the long video, obtain a motion vector of each video frame;

[0070] In this embodiment, the motion vector is calculated by the optical flow method or the block matching method. When the optical flow method is selected for calculation, the optical flow field between video frames is calculated to obtain the motion vector of each pixel; when the block matching method is selected for calculation, the video frame is divided into blocks of the same size, and then the block most similar to the current block is searched in the adjacent video frames to calculate the motion vector of each block. At the same time, the mean square error and absolute error methods can be used to measure the similarity of the blocks.

[0071] S2122: Based on the acquired motion vector differences under adjacent video frames, a preset abnormality threshold is compared to determine a motion abnormality area;

[0072] In this embodiment, when the optical flow method is selected for comparison to determine the abnormal motion area, the difference between the size or direction of the pixel motion vector and the surrounding pixels is calculated and compared with the preset abnormal threshold. When the calculated difference exceeds the preset abnormal threshold, it is determined that the motion under the current pixel is abnormal; when the block matching method is selected for comparison to determine the abnormal motion area, the difference between adjacent blocks is calculated and compared with the preset abnormal threshold. When the calculated difference exceeds the preset abnormal threshold, it is determined that the motion under the current block is abnormal.

[0073] S2123: performing noise reduction processing based on the abnormal motion area to obtain a corrected video frame;

[0074] In this embodiment, when a motion abnormality region is obtained based on abnormal pixels, a median filter, a mean filter or a Gaussian filter is used to replace the motion vector of the abnormal pixel with the average value or median value of the motion vectors of its surrounding pixels to perform noise reduction processing; when a motion abnormality region is obtained based on an abnormal block, a 3D noise reduction algorithm is used to filter the pixels of the abnormal block in combination with the current video frame and the adjacent video frames to perform noise reduction processing.

[0075] S2124: Based on the corrected video frame, the target object is used as a detection subject to perform target object feature recognition.

[0076] S213: Taking the target object as the detection subject, a third-party platform recognition processing model is used to detect the long video clips that need to be edited.

[0077] Recognize the target object video clip in the video;

[0078] S2131: Load a long video to be recognized and decode it to obtain multiple pictures of continuous video frames;

[0079] S2132: Using AI vision algorithm to identify target objects in pictures of multiple consecutive video frames;

[0080] S2133: continuously tracking the target object using a target tracking algorithm, and associating adjacent video frame images containing the target object to obtain a video clip containing the target object, and performing feature recognition of the target object;

[0081] S22: based on the identified target object video segment, confirming the start time and the end time of the target object video segment;

[0082] S221: when a video frame containing a target object is detected and identified, searching backward based on the time point of the current video frame until the target object is no longer detected and identified, and recording the time point when the target object position is no longer detected and identified as the start time;

[0083] S222: When a video frame containing a target object is detected and identified, continue searching forward based on the time point of the current video frame until the target object is no longer detected and identified, and record the time point when the target object's video position is no longer detected and identified as the end time.

[0084] S23: Based on the target object video clip, identify the target object's wonderful content, and

[0085] The video clip is determined to be a wonderful video clip of the target object;

[0086] S231: based on the target object video clip, when the calculated motion amount difference between the current video frame and the adjacent video frame exceeds a preset motion amount threshold, defining the current video frame as the first wonderful content;

[0087] In this embodiment, it is considered that the area with a large amount of motion represents an exciting action scene, and the current video frame is defined as exciting content. It should be noted that the calculation of the amount of motion of the target object can be selected from the optical flow method and the block matching method.

[0088] S232: extracting the audio track of the target video clip, performing audio feature analysis, and defining the current video frame as the second exciting content when the calculated audio feature difference between the current video frame and the adjacent video frame exceeds a preset audio feature threshold, or the audio feature value of the current video frame is within a preset audio feature range;

[0089] In this embodiment, when the audio energy suddenly increases or changes within a preset audio feature threshold range, an explosion or cheering may occur, that is, there is a wonderful moment, and the current video frame is defined as a wonderful content.

[0090] S233: when the current video frame belongs to both the first wonderful content and the second wonderful content, determining the target object video segment where the current video frame is located as the target object wonderful video segment;

[0091] S3: based on the start time and end time of the wonderful video clip of the target object, the long video is clipped to obtain the target object slice video;

[0092] S4: Merge multiple sliced ​​videos of the target object in chronological order to form a video, and output the highlight clips of the target object.

[0093] Example 2

[0094] Based on the same design concept, this embodiment provides a method for extracting wonderful clips from a long video, specifically:

[0095] like Figure 2 As shown, when the openvc recognition processing model is used for recognition processing, it specifically includes:

[0096] T11: Input multiple pictures containing target objects and long videos to be edited, and perform feature extraction on the pictures containing target objects;

[0097] T12: Based on the long video to be edited, the binary method is used to extract video frames as key frames in a timely manner;

[0098] T13: Taking the target object as the detection subject, the target object feature recognition is performed to obtain the target object video clip;

[0099] T131: When the detection recognizes a video frame with a target object, the search is backtracked based on the time point of the current video frame until the detection fails to recognize the target object, and the time point at which the detection fails to recognize the video position of the target object is set as

[0100] Record as the start time;

[0101] T132: When the detection recognizes a video frame with a target object, continue searching forward based on the time point of the current video frame until the detection cannot recognize the target object, and the time point at which the detection cannot recognize the video position of the target object is set as

[0102] Record as the end time;

[0103] T133: Based on the start time and the end time, obtain the video clip of the target object;

[0104] T14: Based on the video clips of the target object, identify the wonderful video clips of the target object;

[0105] T141: Perform motion analysis based on adjacent key frames, calculate the motion difference between adjacent key frames, and compare it with the preset

[0106] Compare the motion threshold to determine the first exciting content segment;

[0107] T1411: when the difference in motion between adjacent key frames exceeds a preset motion threshold, determining that the video segment where the current key frame is located is the first wonderful content segment;

[0108] T1412: when the difference in motion between adjacent key frames does not exceed a preset motion threshold, it is considered that the video segment where the current key frame is located is not the first wonderful content segment;

[0109] T142: Based on the first wonderful content segment, extract the audio track of the video, perform audio feature analysis, calculate the audio features, and compare them with the preset audio feature threshold to determine the second wonderful content segment as the final target object

[0110] Highlight video clips;

[0111] T1421: when the audio feature is greater than a preset audio feature threshold, or there is an audio feature in a specific frequency range, determining that the first wonderful content segment where the current key frame is located is the second wonderful content segment, and using it as the final target object wonderful video segment;

[0112] T1422: When the audio feature is not greater than a preset audio feature threshold, or there is an audio feature in a specific frequency range, it is considered that the first wonderful content segment where the current key frame is located is not the second wonderful content segment;

[0113] T15: Based on the start and end times of the wonderful video clips of the target object, the long video is clipped to obtain the target object slice video;

[0114] T16: Combine multiple sliced ​​videos of the target object in chronological order to form a video, and output the highlight clips of the target object.

[0115] like Figure 3 As shown, when the deep learning recognition processing module is used for recognition processing, it specifically includes:

[0116] T21: Input multiple pictures containing target objects and long videos to be edited, and perform feature extraction on the pictures containing target objects;

[0117] T22: Based on the long video that needs to be edited, a deep learning processing model is used for noise reduction;

[0118] T221: Use deep learning algorithm for noise reduction;

[0119] T2211: Collect a large amount of noisy images or video data, as well as the corresponding clean images or video data without noise, for deep learning model training;

[0120] T2212: Build a deep learning model, use noisy images or video data as the input of the deep learning model, and the corresponding clean images or video data without noise as the output of the deep learning model, and perform model denoising training;

[0121] In this embodiment, the deep learning model includes a convolutional neural network model and a recurrent neural network model.

[0122] T2213: Use the loss function to measure the difference between the output of the deep learning model and the true value, adjust the model parameters, and optimize the model by minimizing the loss function to obtain the trained deep learning model;

[0123] In this embodiment, the loss function includes a mean square error loss function and a cross entropy loss function. The loss function is minimized by adjusting the model parameters to improve the model performance.

[0124] T2214: Input the long video to be edited into the trained deep learning model and remove the noise from the output

[0125] The long video after

[0126] T222: Use spatiotemporal filtering method for noise reduction;

[0127] T2221: Based on the long video to be edited, perform spatial domain filtering on each video frame to obtain a long video after removing spatial noise;

[0128] In this embodiment, the spatial domain filtering processing method includes median filtering, mean filtering, and Gaussian filtering.

[0129] T2222: Based on the long video after removing the spatial noise, the information between adjacent video frames is used to perform time domain filtering to obtain the long video after removing the temporal noise, that is, to obtain the long video after removing the temporal and spatial noise;

[0130] In this embodiment, the method of using information between adjacent video frames to perform time domain filtering includes using a video frame difference method to analyze the pixel value differences between adjacent video frames to remove false motion information caused by noise.

[0131] T223: Use motion compensation method for noise reduction;

[0132] T2231: Based on the long video to be edited, perform motion estimation on each video frame to obtain the motion vector of the target object;

[0133] T2232: Based on the motion vector of the target object, motion compensation is performed on the video frame to predict the position of the target object in the current video frame to obtain a compensated video frame;

[0134] In this embodiment, a bilinear interpolation or bicubic interpolation algorithm is used to fill in the missing pixel values ​​to obtain a compensated video frame.

[0135] T2233: Based on the compensated video frames, use deep learning algorithms or spatiotemporal filtering methods to perform noise reduction

[0136] Processing, get the long video after removing the noise;

[0137] T23: Taking the target object as the detection subject, the target object video clip recognition is performed;

[0138] T231: When the detection recognizes a video frame with a target object, the search is backtracked based on the time point of the current video frame until the detection fails to recognize the target object, and the time point at which the detection fails to recognize the video position of the target object is set as

[0139] Record as the start time;

[0140] T232: When the detection recognizes a video frame with a target object, continue searching forward based on the time point of the current video frame until the detection cannot recognize the target object, and the time point at which the detection cannot recognize the video position of the target object is set as

[0141] Record as the end time;

[0142] T233: Based on the start time and the end time, obtain the video clip of the target object;

[0143] T24: Based on the video clips of the target object, identify the wonderful video clips of the target object;

[0144] T241: Perform motion analysis based on adjacent key frames, calculate the motion of adjacent key frames, and compare it with the preset motion

[0145] The quantity threshold is compared to determine the first wonderful content segment;

[0146] T2411: when the motion amount of adjacent key frames exceeds a preset motion amount threshold, determining that the video segment where the current key frame is located is the first wonderful content segment;

[0147] T2412: When the motion amount of adjacent key frames does not exceed the preset motion amount threshold, it is considered that the video segment where the current key frame is located is not the first wonderful content segment;

[0148] T242: Based on the first wonderful content segment, extract the audio track of the video, perform audio feature analysis, calculate the audio features, and compare them with the preset audio feature threshold to determine the second wonderful content segment as the final target object

[0149] Highlight video clips;

[0150] T2421: when the audio feature is greater than a preset audio feature threshold, or there is an audio feature in a specific frequency range, determining that the first wonderful content segment where the current key frame is located is the second wonderful content segment, and using it as the final target object wonderful video segment;

[0151] T2422: When the audio feature is not greater than a preset audio feature threshold, or there is an audio feature in a specific frequency range, it is considered that the first wonderful content segment where the current key frame is located is not the second wonderful content segment;

[0152] T25: Based on the start and end times of the wonderful video clips of the target object, the long video is clipped to obtain the target object slice video;

[0153] T26: Combine multiple sliced ​​videos of the target object in chronological order to form a video, and output the highlight clips of the target object.

[0154] like Figure 4 As shown, when the third-party platform identification processing module is used for identification processing, it specifically includes:

[0155] T31: Input multiple pictures containing target objects and long videos to be edited, and perform feature extraction on the pictures containing target objects;

[0156] T32: Load the long video to be recognized and decode it to obtain multiple pictures of continuous video frames;

[0157] T33: Uses AI vision algorithms to identify target objects in pictures of multiple consecutive video frames;

[0158] T34: Use the target tracking algorithm to continuously track the target object, associate the adjacent video frames containing the target object, obtain the video clip containing the target object, and perform target object feature recognition;

[0159] T35: Take the target object as the detection subject and perform target object video segment recognition;

[0160] T351: When the detection recognizes a video frame with a target object, the search is backtracked based on the time point of the current video frame until the detection fails to recognize the target object, and the time point at which the detection fails to recognize the video position of the target object is set as

[0161] Record as the start time;

[0162] T352: When the detection recognizes a video frame with a target object, continue searching forward based on the time point of the current video frame until the detection cannot recognize the target object, and the time point at which the detection cannot recognize the video position of the target object is set as

[0163] Record as the end time;

[0164] T353: Get a video clip of the target object based on the start time and the end time;

[0165] T36: Based on the video clips of the target object, identify the wonderful video clips of the target object;

[0166] T361: Perform motion analysis based on adjacent key frames, calculate the motion of adjacent key frames, and compare it with the preset motion

[0167] The quantity threshold is compared to determine the first wonderful content segment;

[0168] T3611: when the motion amount of adjacent key frames exceeds a preset motion amount threshold, determining that the video segment where the current key frame is located is the first wonderful content segment;

[0169] T3612: When the motion amount of adjacent key frames does not exceed the preset motion amount threshold, it is considered that the video segment where the current key frame is located is not the first wonderful content segment;

[0170] T362: Based on the first wonderful content segment, extract the audio track of the video, perform audio feature analysis, calculate the audio features, and compare them with the preset audio feature threshold to determine the second wonderful content segment as the final target object

[0171] Highlight video clips;

[0172] T3621: when the audio feature is greater than a preset audio feature threshold, or there is an audio feature in a specific frequency range, determining that the first wonderful content segment where the current key frame is located is the second wonderful content segment, and using it as the final target object wonderful video segment;

[0173] T3622: When the audio feature is not greater than a preset audio feature threshold, or there is an audio feature in a specific frequency range, it is considered that the first wonderful content segment where the current key frame is located is not the second wonderful content segment;

[0174] T37: Based on the start time and end time of the wonderful video clip of the target object, the long video is clipped to obtain the target object slice video;

[0175] T38: Combine multiple sliced ​​videos of the target object in chronological order to form a video, and output the highlight clips of the target object.

[0176] Example 3

[0177] Based on the same design concept, Figure 5 As shown, this embodiment provides a system for extracting wonderful clips from a long video, including an input and pre-processing module, a recognition processing module, a cropping module, and an integrated output module.

[0178] Specifically, the input and preprocessing processing module is used to input multiple pictures containing target objects and long videos that need to be edited, and perform feature extraction on the pictures containing target objects;

[0179] The recognition processing module is used to identify the target object video clips in the long video to be edited, and confirm the start and end time of the target object's wonderful video clips;

[0180] A cropping module is used to crop the long video based on the start time and end time of the target object's wonderful video clip to obtain the target object slice video;

[0181] The integrated output module is used to merge multiple sliced ​​videos of the target object into one video in chronological order and output the highlight clips of the target object.

[0182] Example 4

[0183] Based on the same technical concept, the embodiment of the present application also provides a computer device, including a memory 1 and a processor 2, such as Figure 6 As shown, the memory 1 stores a computer program, and the processor 2 implements any of the above-mentioned methods when executing the computer program.

[0184] Among them, the memory 1 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 1 can be an internal storage unit of the system for extracting highlights from long videos, such as a hard disk. In other embodiments, the memory 1 can also be an external storage device of the system for extracting highlights from long videos, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 1 can also include both an internal storage unit and an external storage device of the system for extracting highlights from long videos. The memory 1 can not only be used to store application software and various types of data installed in the system for extracting highlights from long videos, such as the code of the system program for extracting highlights from long videos, but can also be used to temporarily store data that has been output or is to be output.

[0185] In some embodiments, the processor 2 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, which is used to run the program code stored in the memory 1 or process data, such as executing a system program for extracting wonderful clips from a long video.

[0186] The disclosed embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the steps of the method described in the above method embodiment. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0187] The computer program product of the application page content refresh method provided in the disclosed embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0188] The disclosed embodiment of the present invention also provides a computer program, which implements any one of the methods of the aforementioned embodiments when executed by a processor. The computer program product can be implemented in hardware, software or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0189] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0190] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0191] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0192] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0193] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0194] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0195] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0196] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0197] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for extracting wonderful clips from a long video, characterized in that: include: Input multiple pictures containing target objects and long videos to be edited, and perform feature extraction on the pictures containing target objects; Taking the target object as the detection subject, identify the target object video clips in the long video that needs to be edited, and confirm the start and end time of the target object's wonderful video clips; Based on the start and end times of the wonderful video clips of the target object, the long video is clipped to obtain the sliced ​​video of the target object; Combine multiple sliced ​​videos of the target object in chronological order to form a video and output the highlight clips of the target object.

2. The method for extracting highlights from a long video according to claim 1, characterized in that: The method of taking the target object as the detection subject, identifying the wonderful video clips of the target object in the long video to be edited, and confirming the start time and end time of the wonderful video clips of the target object includes: Taking the target object as the detection subject, the target object features are identified in the long video to be edited, and the start and end times of the target object video clip are confirmed; Based on the target object video clip, the target object wonderful content is identified, and the video clip where the target object wonderful content is located is determined as the target object wonderful video clip.

3. The method for extracting highlights from a long video according to claim 2, characterized in that: The target object video segment recognition in the long video to be edited includes using an opencv recognition processing model to recognize the target object video segment in the long video to be edited, specifically: Based on the timeline of the long video, the long video to be edited is divided into blocks in an orderly manner to obtain multiple block videos; Based on multiple block videos, binary division is used to extract video frames as key frames in a timely manner; The target object is taken as the detection subject, and the target object features are recognized on the key frames.

4. The method for extracting highlights from a long video according to claim 2, characterized in that: The identifying of the target object video segment in the long video to be edited also includes using a deep learning recognition processing model to identify the target object video segment in the long video to be edited, specifically including: Based on the video frames of the long video, obtain the motion vector of each video frame; Based on the acquired motion vector differences under adjacent video frames, a preset abnormality threshold is compared to determine the motion abnormality area; Based on the abnormal motion area, noise reduction processing is performed to obtain a corrected video frame; Based on the corrected video frame, the target object is taken as the detection subject to perform target object feature recognition.

5. The method for extracting highlights from a long video according to claim 2, characterized in that: The identifying of the target object video segment in the long video to be edited also includes using a third-party platform recognition processing model to identify the target object video segment in the long video to be edited, specifically: Load the long video to be recognized and decode it to obtain multiple pictures of continuous video frames; Use AI vision algorithms to identify target objects in pictures of multiple consecutive video frames; The target tracking algorithm is used to continuously track the target object, and the adjacent video frame images containing the target object are associated to obtain the video clip containing the target object, and the target object features are recognized.

6. The method for extracting highlights from a long video according to claim 2, characterized in that: The step of confirming the start time and end time of the target object video clip includes: When a video frame containing a target object is detected and identified, a search is performed backwards based on the time point of the current video frame until the target object is no longer detected and identified, and the time point at which the target object's video position is no longer detected and identified is recorded as the start time; When a video frame containing a target object is detected and identified, the search continues forward based on the time point of the current video frame until the target object is no longer detected and identified, and the time point when the target object's video position is no longer detected and identified is recorded as the end time.

7. The method for extracting highlights from a long video according to claim 2, characterized in that: The method of identifying the wonderful content of the target object based on the target object video clip and determining the video clip containing the wonderful content of the target object as the wonderful video clip of the target object includes: Based on the target object video clip, when the calculated motion difference between the current video frame and the adjacent video frame exceeds a preset motion threshold, the current video frame is defined as the first wonderful content; Extract the audio track of the target object video clip, perform audio feature analysis, and define the current video frame as the second exciting content when the calculated audio feature difference between the current video frame and the adjacent video frame exceeds a preset audio feature threshold, or the audio feature value of the current video frame is within a preset audio feature range; When the current video frame belongs to both the first wonderful content and the second wonderful content, the target object video segment where the current video frame is located is determined as the target object wonderful video segment.

8. A system for extracting highlights from a long video according to claim 1, characterized in that: include: An input and preprocessing module, which is used to input multiple pictures containing target objects and long videos that need to be edited, and perform feature extraction on the pictures containing target objects; The recognition processing module is used to identify the target object video clips in the long video to be edited, and confirm the start and end time of the target object's wonderful video clips; A cropping module is used to crop the long video based on the start time and end time of the target object's wonderful video clip to obtain the target object slice video; The integrated output module is used to merge multiple sliced ​​videos of the target object into one video in chronological order and output the highlight clips of the target object.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images