Video processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a video processing method, device, electronic equipment and storage medium. The method comprises: obtaining a plurality of continuous to-be-processed video frames from a target video; the number of the plurality of continuous to-be-processed video frames is less than the number of video frames contained in the target video; performing feature extraction on the plurality of continuous to-be-processed video frames to obtain video frame feature information; and performing real-time key frame identification based on the video frame feature information to determine a key frame identification result corresponding to the plurality of continuous to-be-processed video frames. The present disclosure can improve the real-time performance and efficiency of key frame identification.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a video processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Keyframe recognition is an important problem in the field of computer vision. It has important applications in many fields, such as video cover selection and editing of highlights.

[0003] In related technologies, identifying keyframes from videos takes a long time, resulting in low real-time performance of keyframe recognition; furthermore, the definition of keyframes is universal and there is no separate definition for different video scenarios. Summary of the Invention

[0004] This disclosure provides a video processing method, apparatus, electronic device, and storage medium, which can improve the real-time performance of keyframe recognition and the accuracy of keyframe recognition in jumping scenes. The technical solution of this disclosure is as follows:

[0005] According to an embodiment of this disclosure, a video processing method is provided, including:

[0006] Multiple consecutive video frames to be processed are obtained from the target video; the number of the multiple consecutive video frames to be processed is less than the number of video frames contained in the target video.

[0007] Feature extraction is performed on the multiple consecutive video frames to be processed to obtain video frame feature information;

[0008] Based on the video frame feature information, keyframes are identified in real time to determine the keyframe identification results corresponding to the multiple consecutive video frames to be processed.

[0009] In the above technical solution, by acquiring multiple consecutive video frames to be processed each time, key frames are identified from the multiple consecutive video frames to be processed. Since the number of multiple consecutive video frames to be processed is less than the number of video frames contained in the target video, the amount of data to be processed each time key frame recognition can be reduced, thereby improving the real-time performance and efficiency of key frame recognition.

[0010] In an optional embodiment, the step of extracting features from the plurality of consecutive video frames to be processed to obtain video frame feature information includes:

[0011] Based on the feature extraction model, feature extraction is performed on the multiple consecutive video frames to be processed to obtain the feature information of the video frames;

[0012] The method further includes a method for training the feature extraction model, the training method of the feature extraction model including:

[0013] Obtain processed sample video frames; the processed sample video frames are obtained by performing image perturbation processing on the original sample video frames.

[0014] The preset machine learning model is trained based on the processed sample video frames to obtain the feature extraction model.

[0015] In the above technical solution, when training the feature extraction model, image perturbation processing is performed on the sample video frames, so that the feature extraction model can learn more image information in different scenarios, thereby improving the adaptability and generalization ability of the feature extraction model.

[0016] In an optional embodiment, the method includes an image perturbation processing method, the image perturbation processing method comprising:

[0017] Pixel perturbation processing is performed on each pixel of the original sample video frame to obtain a pixel-perturbed video frame;

[0018] The processed sample video frame is generated based on the pixel-perturbed video frame.

[0019] In the above technical solution, by perturbing the pixels of the sample video frame, the pixel values ​​in the sample video frame can be changed to obtain the perturbed video frame. This perturbation method is simple to operate, easy to implement, and can improve the efficiency of image perturbation.

[0020] In an optional embodiment, the original sample video frames comprise multiple consecutive sample video frames; the image perturbation processing method includes:

[0021] The first frame of the plurality of consecutive sample video frames is determined as the reference frame;

[0022] Determine the reference acquisition height of the reference frame; the reference acquisition height represents the acquisition height of the image acquisition device when acquiring the reference frame;

[0023] Based on the reference acquisition height, the target acquisition height of the sample video frames after the reference frame is determined;

[0024] Based on the target acquisition height, image transformation is performed on the sample video frames after the reference frame to obtain transformed video frames;

[0025] The processed sample video frame is generated based on the reference frame and the transformed video frame.

[0026] In the above technical solution, adjusting the acquisition height can simulate the rising or falling state of the acquisition lens, thereby perturbing the image from the spatiotemporal sequence of image acquisition, realizing the diversity of image perturbation and improving the effect of image perturbation.

[0027] In an optional embodiment, the step of performing real-time keyframe identification based on the video frame feature information to determine the keyframe identification results corresponding to the plurality of consecutive video frames to be processed includes:

[0028] The video feature information is input into the target classification model, and the video frame to be classified is classified based on the target classification model to obtain the target classification result of the video frame to be classified; the video frame to be classified is any video frame other than the first frame and the last frame among the multiple consecutive video frames to be processed.

[0029] Based on the target classification result, the keyframe recognition result is determined.

[0030] In the above technical solution, the video frame to be classified is the intermediate frame of multiple consecutive video frames. Thus, the video frames before and after the intermediate frame can provide a basis for classification. That is, by using the preceding and following frames as a reference for classifying the video frame to be classified, the classification accuracy can be improved.

[0031] In an optional embodiment, classifying the video frame to be classified based on the target classification model to obtain the target classification result for the video frame to be classified includes:

[0032] Based on the target classification model, feature analysis is performed on the video feature information corresponding to each of the multiple consecutive video frames to be processed to obtain feature analysis results; the feature analysis results characterize the jumping action in the video frame to be classified and the association information with the jumping action in the first video frame; the first video frame is the video frame other than the video frame to be classified among the multiple consecutive video frames to be processed.

[0033] The video frames to be classified are classified based on the association information to obtain the target classification result.

[0034] In the above technical solution, in jumping scenarios, feature analysis can be performed on the feature information of video frames to obtain the correlation information between jumping actions in the video frame to be classified and jumping actions in other video frames. This makes it easier to determine whether the jumping action in the video frame to be classified meets the jumping conditions, thereby further improving the accuracy of video frame classification.

[0035] In an optional embodiment, determining the keyframe recognition result based on the target classification result includes:

[0036] If the target classification result indicates that the jump height of the jumping action in the video frame to be classified is greater than the jump height of the jumping action in the first video frame, then the video frame to be classified is identified as a keyframe.

[0037] In the above technical solution, the key frame is determined by comparing the jump height of the jumping action in the video frame to be classified with the jump height of the jumping action in the first video frame. Since the jump height can be directly obtained based on the analysis of video frame feature information, the convenience and efficiency of key frame determination can be improved.

[0038] In an optional embodiment, the method includes a method for training the target classifier, the training method comprising:

[0039] The video feature information corresponding to each of the multiple consecutive sample video frames is input into a first preset classifier to obtain the first classification information for the second video frame; the second video frame is any video frame among the multiple consecutive sample video frames except for the first frame and the last frame; the first classification information represents the probability that the jumping action in the second video frame satisfies the preset jumping condition.

[0040] The video feature information corresponding to each of the multiple consecutive sample video frames is input into the second preset classifier to obtain the second classification information of the second video frame; the second classification information represents the probability that there is jump judgment auxiliary information in the second video frame.

[0041] First backpropagation gradient information is generated based on the first classification information, and second backpropagation gradient information is generated based on the second classification information;

[0042] Based on the first backpropagation gradient information and the second backpropagation gradient information, the first preset classifier is trained to obtain the target classifier.

[0043] In the above technical solution, when training the target classifier, the model is trained based on the back gradient information of the first preset classifier and the back gradient information of the auxiliary second preset classifier to obtain the target classifier, so that the target classifier can obtain more auxiliary training information, thereby improving the classification performance of the target classifier.

[0044] In an optional embodiment, the target classifier includes multiple target sub-classifiers, and the method further includes a method for training the multiple sub-classifiers, the training method including:

[0045] The video feature information corresponding to each of the multiple consecutive sample video frames is input into multiple preset sub-classifiers to obtain the classification output information of the multiple preset sub-classifiers for the third video frame; the third video frame is any video frame in the multiple consecutive sample video frames except for the first frame and the last frame; the classification output information represents the probability that the jumping action in the third video frame satisfies the preset jumping condition.

[0046] Based on the classification output information corresponding to each of the multiple preset sub-classifiers, the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers is generated.

[0047] Based on the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers, the multiple preset sub-classifiers are trained to obtain the multiple target sub-classifiers.

[0048] The process of classifying the video frames to be classified based on the target classification model to obtain the target classification result for the video frames to be classified includes:

[0049] The video frames to be classified are classified based on the multiple target sub-classifiers respectively, and the classification results corresponding to each of the multiple target sub-classifiers are obtained.

[0050] The classification results of the multiple target sub-classifiers are fused to obtain the target classification result.

[0051] In the above technical solution, the back gradient information of multiple sub-classifiers during the training process is mutually supportive, thereby enabling multiple sub-classifiers to assist in training and improve the classification performance of multiple sub-classifiers; furthermore, by fusing the classification results of multiple sub-classifiers to determine the target classification result, the accuracy of the target classification result can be improved.

[0052] On the other hand, according to embodiments of this disclosure, a video processing apparatus is also provided, comprising:

[0053] The first acquisition module is used to acquire multiple consecutive video frames to be processed from the target video; the number of the multiple consecutive video frames to be processed is less than the number of video frames contained in the target video.

[0054] The feature extraction module is used to extract features from the multiple consecutive video frames to be processed to obtain video frame feature information;

[0055] The keyframe recognition module is used to perform real-time keyframe recognition based on the video frame feature information, and to determine the keyframe recognition results corresponding to the multiple consecutive video frames to be processed.

[0056] In an optional embodiment, the feature extraction module includes:

[0057] The feature information determination module is used to extract features from the multiple consecutive video frames to be processed based on the feature extraction model to obtain the feature information of the video frames;

[0058] The device further includes:

[0059] The second acquisition module is used to acquire processed sample video frames; the processed sample video frames are obtained by performing image perturbation processing on the original sample video frames.

[0060] The first training module is used to train a preset machine learning model based on the processed sample video frames to obtain the feature extraction model.

[0061] In an optional embodiment, the apparatus further includes:

[0062] The pixel perturbation module is used to perform pixel perturbation processing on each pixel of the original sample video frame to obtain a pixel-perturbated video frame.

[0063] The first generation module is used to generate the processed sample video frame based on the pixel-perturbed video frame.

[0064] In an optional embodiment, the original sample video frames comprise a plurality of consecutive sample video frames; the apparatus further comprises:

[0065] The first determining module is used to determine the first frame of the plurality of consecutive sample video frames as the reference frame;

[0066] The second determining module is used to determine the reference acquisition height of the reference frame; the reference acquisition height represents the acquisition height of the image acquisition device when acquiring the reference frame;

[0067] The acquisition height determination module is used to determine the target acquisition height of sample video frames after the reference frame based on the reference acquisition height.

[0068] The image transformation module is used to perform image transformation on the sample video frames after the reference frame based on the target acquisition height to obtain transformed video frames;

[0069] The second generation module is used to generate the processed sample video frame based on the reference frame and the transformed video frame.

[0070] In an optional embodiment, the keyframe recognition module includes:

[0071] The first classification module is used to input the video feature information into the target classification model, classify the video frame to be classified based on the target classification model, and obtain the target classification result of the video frame to be classified; the video frame to be classified is any video frame other than the first frame and the last frame among the multiple consecutive video frames to be processed.

[0072] The third determining module is used to determine the keyframe recognition result based on the target classification result.

[0073] In an optional embodiment, the first classification module includes:

[0074] The feature analysis module is used to perform feature analysis on the video feature information corresponding to each of the multiple consecutive video frames to be processed based on the target classification model, and obtain feature analysis results; the feature analysis results characterize the jumping action in the video frame to be classified and the association information with the jumping action in the first video frame; the first video frame is a video frame other than the video frame to be classified among the multiple consecutive video frames to be processed.

[0075] The second classification module is used to classify the video frames to be classified based on the association information to obtain the target classification result.

[0076] In an optional embodiment, the third determining module includes:

[0077] The fourth determining module is used to determine the video frame to be classified as a keyframe when the target classification result indicates that the jump height of the jumping action in the video frame to be classified is greater than the jump height of the jumping action in the first video frame.

[0078] In an optional embodiment, the device includes:

[0079] The first input module is used to input the video feature information corresponding to each of the plurality of consecutive sample video frames into the first preset classifier to obtain the first classification information of the second video frame; the second video frame is any video frame other than the first frame and the last frame among the plurality of consecutive sample video frames; the first classification information represents the probability that the jumping action in the second video frame satisfies the preset jumping condition.

[0080] The second input module is used to input the video feature information corresponding to each of the multiple consecutive sample video frames into the second preset classifier to obtain the second classification information of the second video frame; the second classification information represents the probability that there is jump judgment auxiliary information in the second video frame.

[0081] The third generation module is used to generate first backpropagation gradient information based on the first classification information and to generate second backpropagation gradient information based on the second classification information.

[0082] The second training module is used to train the first preset classifier based on the first backpropagation gradient information and the second backpropagation gradient information to obtain the target classifier.

[0083] In an optional embodiment, the target classifier includes multiple target sub-classifiers, and the apparatus further includes:

[0084] The third input module is used to input the video feature information corresponding to each of the multiple consecutive sample video frames into multiple preset sub-classifiers to obtain the classification output information of the multiple preset sub-classifiers for the third video frame; the third video frame is any video frame in the multiple consecutive sample video frames except for the first frame and the last frame; the classification output information represents the probability that the jumping action in the third video frame satisfies the preset jumping condition.

[0085] The fourth generation module is used to generate the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers based on the classification output information corresponding to each of the multiple preset sub-classifiers.

[0086] The third training module is used to train the multiple preset sub-classifiers based on the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers, so as to obtain the multiple target sub-classifiers.

[0087] The first classification module includes:

[0088] The third classification module is used to classify the video frame to be classified based on the multiple target sub-classifiers respectively, and obtain the classification results corresponding to each of the multiple target sub-classifiers.

[0089] The information fusion module is used to fuse the classification results of the multiple target sub-classifiers to obtain the target classification result.

[0090] On the other hand, an electronic device is also provided according to embodiments of this disclosure, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in any of the preceding claims.

[0091] On the other hand, according to embodiments of the present disclosure, a computer-readable storage medium is also provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the methods described above in the embodiments of the present disclosure.

[0092] On the other hand, according to embodiments of the present disclosure, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described above in embodiments of the present disclosure.

[0093] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0094] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0095] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment.

[0096] Figure 2 This is a flowchart illustrating a video processing method according to an exemplary embodiment.

[0097] Figure 3 This is a flowchart illustrating a feature extraction model training method according to an exemplary embodiment.

[0098] Figure 4 This is a flowchart illustrating an image perturbation method according to an exemplary embodiment.

[0099] Figure 5 This is a flowchart illustrating another image perturbation method according to an exemplary embodiment.

[0100] Figure 6 This is a flowchart illustrating a keyframe determination method according to an exemplary embodiment.

[0101] Figure 7 This is a flowchart illustrating a method for classifying video frames to be classified according to an exemplary embodiment.

[0102] Figure 8 This is a flowchart illustrating a training method for a target classifier according to an exemplary embodiment.

[0103] Figure 9 This is a flowchart illustrating another target classifier training method according to an exemplary embodiment.

[0104] Figure 10 This is a schematic diagram of a keyframe recognition model according to an exemplary embodiment.

[0105] Figure 11 This is a schematic diagram of a video processing apparatus according to an exemplary embodiment.

[0106] Figure 12 This is a block diagram illustrating an electronic device for video processing according to an exemplary embodiment.

[0107] Figure 13 This is a block diagram illustrating another electronic device for video processing according to an exemplary embodiment. Detailed Implementation

[0108] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0109] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0110] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include server 100 and terminal 200.

[0111] In one optional embodiment, server 100 can be used to generate a video processing model related to video processing; when receiving a video processing request sent by terminal 200, it processes the video to be processed based on the video processing model to obtain keyframes in the video to be processed. Specifically, server 100 can be a standalone server, a server cluster consisting of multiple servers, or a distributed system.

[0112] In an optional embodiment, terminal 200 can combine the video processing model generated by server 100 to directly process the video to be processed, obtaining keyframes from the video. Specifically, terminal 200 can be, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices. Optionally, the operating system running on the electronic device can be, but is not limited to, Android, iOS, Linux, and Windows.

[0113] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by this disclosure. In practical applications, other application environments may also be included, such as video processing models that can also be implemented in terminal 200.

[0114] In the embodiments described in this specification, the server 100 and the terminal 200 can be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any restrictions.

[0115] Figure 2 This is a flowchart illustrating a video processing method according to an exemplary embodiment, such as... Figure 2 As shown, this method can be applied to electronic devices such as terminals, servers, and edge computing nodes, and may include:

[0116] S210. Obtain multiple consecutive video frames to be processed from the target video; the number of the multiple consecutive video frames to be processed is less than the number of video frames contained in the target video.

[0117] In this embodiment, the multiple consecutive video frames to be processed can be video frames that are consecutive in time. These multiple consecutive video frames to be processed can be video frames obtained from real-time data streams or video frames from pre-stored preset videos.

[0118] Specifically, when multiple consecutive video frames to be processed are video frames obtained from a real-time data stream, in a live streaming scenario, the real-time data stream can be a live data stream pulled from a live streaming server; in a real-time shooting scenario, the real-time data stream can be a video data stream continuously acquired and recorded by an image acquisition device. When multiple consecutive video frames to be processed are video frames from a stored preset video, the preset video can be pre-divided into frames to obtain multiple consecutive video frames, and then multiple consecutive video frames to be processed can be obtained from the divided video frames.

[0119] In another optional embodiment, the target video can be a video that has already been completed and stored, or it can be a video that is being broadcast live or recorded. For example, if multiple consecutive video frames to be processed are video frames from a stored preset video, the target video can be the preset video; if multiple consecutive video frames to be processed are video frames obtained from a real-time data stream, the target video can be the final video formed during the entire live broadcast process, or it can be the video formed after real-time recording is completed.

[0120] Therefore, the target video can generally include a large number of video frames. In this embodiment, the number of multiple consecutive video frames to be processed each time is less than the number of video frames contained in the target video. This means that by obtaining a small number of consecutive video frames to be processed from the target video and processing a small number of video frames to be processed each time, instead of processing all the video frames contained in the target video at once, the video processing efficiency and real-time performance can be improved.

[0121] S220. Perform feature extraction on the multiple consecutive video frames to be processed to obtain video frame feature information.

[0122] In an optional embodiment, before feature extraction from multiple consecutive video frames to be processed, the video frames to be processed can be preprocessed. Specifically, image enhancement processing can be performed on multiple consecutive video frames to be processed. For example, the image enhancement processing method can include contrast adjustment, brightness adjustment, saturation adjustment, etc., so as to highlight the image features of each video frame to be processed and facilitate feature extraction.

[0123] In another optional embodiment, before feature extraction from multiple consecutive video frames to be processed, preprocessing can be performed on the video frames. Specifically, the following operations can be performed on each of the multiple consecutive video frames to be processed: cropping a preset region around the image center of the video frame; scaling the cropped image to a target size; and performing image feature normalization processing on the image at the target size. Thus, the images after a series of processing correspond to the same measurement benchmark, facilitating unified processing and improving image processing efficiency.

[0124] When extracting features from multiple consecutive video frames to be processed, a feature extraction model with temporal modeling capabilities can be used, or a lightweight feature extraction model can be used. For example, feature extraction models with temporal modeling capabilities can include ResNet, MobilenetV2, etc. Among them, the MobilenetV2 model is lightweight and has a significant advantage in inference time, making it suitable for deployment on mobile devices, edge computing nodes, etc.

[0125] S230. Based on the video frame feature information, perform real-time key frame identification to determine the key frame identification results corresponding to the multiple consecutive video frames to be processed.

[0126] The video processing method provided in this embodiment can identify keyframes in jumping scenes. Keyframes in jumping scenes are related to jumping actions; that is, the jumping action information in keyframes is richer than that in other video frames, and can better represent the uniqueness of the jumping action. Specifically, a video frame in a keyframe whose jumping action satisfies the target jumping condition can be identified as a keyframe.

[0127] Therefore, the above technical solution acquires multiple consecutive video frames to be processed each time and identifies keyframes from these multiple consecutive video frames. Since the number of multiple consecutive video frames to be processed is less than the number of video frames contained in the target video, the amount of data to be processed for each keyframe identification can be reduced, thereby improving the real-time performance and efficiency of keyframe identification. In addition, in jumping scenes, video frames in which the jumping action meets the target jumping conditions are identified as keyframes, thereby improving the targeting and accuracy of keyframe identification in jumping scenes.

[0128] In one embodiment, the number of consecutive video frames to be processed acquired each time can be the same, for example, M; the number of consecutive video frames to be processed acquired each time can be different, for example, M video frames acquired the first time, N video frames acquired the second time, and so on. Each video frame to be processed is acquired at least once. Specifically, the second video frame in the current acquisition of consecutive video frames to be processed can be used as the first video frame in the next acquisition of consecutive video frames to be processed, the third video frame in the current acquisition of consecutive video frames to be processed can be used as the second video frame in the next acquisition of consecutive video frames to be processed, and so on. That is, the position of any video frame other than the first video frame of the target video in the next acquisition of consecutive video frames to be processed is advanced by one position compared with its position in the current acquisition of consecutive video frames to be processed. For example, if the multiple consecutive video frames to be processed acquired in this instance are {frame 1, frame 2, frame 3, frame 4, frame 5}, and the next multiple consecutive video frames to be processed are {frame 2, frame 3, frame 4, frame 5, frame 6}, it can be seen that frame 2 has changed from the 2nd position to the 1st position. Therefore, this video frame acquisition method can achieve maximum coverage of each video frame, avoiding inaccurate recognition results due to incomplete video frame acquisition.

[0129] In another optional embodiment, since some video frames may be repeatedly used as video frames to be processed, repeated feature extraction is not required during feature extraction. Specifically, each video frame can correspond to a corresponding video frame identifier. When a video frame has already undergone feature extraction, the video frame identifier of the video frame will be marked accordingly, and the feature information corresponding to the video frame will be stored. Thus, when the video frame is used as a video frame to be processed again, the feature information corresponding to the video frame can be directly obtained without repeating feature extraction, thereby avoiding the waste of resources caused by repeated feature extraction.

[0130] This embodiment can extract features from the multiple consecutive video frames to be processed based on a feature extraction model to obtain the feature information of the video frames; therefore, please refer to... Figure 3 It illustrates a method for training a feature extraction model, which may include:

[0131] S310. Obtain the processed sample video frame; the processed sample video frame is obtained by performing image perturbation processing on the original sample video frame.

[0132] S320. The preset machine learning model is trained based on the processed sample video frames to obtain the feature extraction model.

[0133] Image perturbation processing refers to image processing methods that transform the original image without affecting its essential features. There can be various image perturbation methods. Each original sample video frame can correspond to one or more processed sample video frames. When there are multiple processed sample video frames, these multiple processed sample video frames can be video frames obtained through different image perturbation methods, or video frames obtained by using one image perturbation method but corresponding to different perturbation parameters. This embodiment does not make specific limitations.

[0134] The video frames after image perturbation processing have the same essential features as the original sample video frames, but may differ in visual effect.

[0135] In the above technical solution, when training the feature extraction model, image perturbation processing is performed on the sample video frames, which enables the feature extraction model to learn more image information in different scenarios, improves the adaptability and generalization ability of the feature extraction model, and enhances the robustness of the model.

[0136] Furthermore, by combining the processed sample video frames obtained after image perturbation with the original sample video frames, sample video frames for training the feature extraction model are obtained, which expands the number of sample video frames and thus improves the model performance of the feature extraction model.

[0137] In one specific embodiment, please refer to Figure 4 It illustrates an image perturbation method, which may include:

[0138] S410. Perform pixel perturbation processing on each pixel of the original sample video frame to obtain a pixel-perturbed video frame.

[0139] S420. Generate the processed sample video frame based on the pixel-perturbed video frame.

[0140] The original sample video frame may contain multiple pixels, and pixel perturbation can be performed based on these pixels to obtain the processed sample video frame.

[0141] In one example, pixel perturbation processing can be pixel blurring. During pixel blurring, when the brightness difference between a pixel and its surrounding pixels in the original sample video frame is less than a first preset value, the brightness of that pixel is smoothed. Pixel blurring can include Gaussian blur, mean blur, median blur, motion blur, etc. The corresponding perturbation parameters can refer to the parameters involved in the pixel blurring process.

[0142] In another example, pixel perturbation processing can be sharpening processing. During sharpening, if the brightness difference between a pixel and its surrounding pixels in the original sample video exceeds a second preset value, the brightness of that pixel is increased. The corresponding perturbation parameters can refer to the parameters involved in the sharpening process.

[0143] In this embodiment, the video frame that has undergone pixel perturbation can be directly identified as the processed sample video frame, or it can be determined by combining other image perturbation methods.

[0144] In the above technical solution, by perturbing the pixels of the sample video frame, the pixel values ​​in the sample video frame can be changed to obtain the perturbed video frame. This perturbation method is simple to operate, easy to implement, and can improve the efficiency of image perturbation.

[0145] In an optional embodiment, the original sample video frames comprise multiple consecutive sample video frames; see also Figure 5 It illustrates another image perturbation method, which may include:

[0146] S510. Determine the first frame of the plurality of consecutive sample video frames as the reference frame.

[0147] S520. Determine the reference acquisition height of the reference frame; the reference acquisition height represents the acquisition height of the image acquisition device when acquiring the reference frame.

[0148] S530. Based on the reference acquisition height, determine the target acquisition height of the sample video frames after the reference frame.

[0149] S540. Based on the target acquisition height, perform image transformation on the sample video frames after the reference frame to obtain transformed video frames.

[0150] S550. Generate the processed sample video frame based on the reference frame and the transformed video frame.

[0151] The number of consecutive sample video frames determined each time can be the same as the number of consecutive video frames to be processed each time, which makes it easier to match the model training process with the model usage process.

[0152] Each video frame has a corresponding acquisition height, which represents the acquisition height of the image acquisition device when acquiring that video frame. Specifically, the acquisition height can be the height of the lens when acquiring the image; the acquisition height can be an absolute height or a relative height. When capturing the target object from above, a higher acquisition height is used, and when capturing the object from below, a lower acquisition height is used. During image acquisition, the acquisition height can be adjusted, which corresponds to adjusting the lens height, thereby simulating the process of the lens rising or falling.

[0153] For multiple consecutive sample video frames, the first frame can be designated as the reference frame. The acquisition height corresponding to the reference frame becomes the reference acquisition height. Therefore, the acquisition height of subsequent sample video frames can be adjusted based on the reference acquisition height. Each sample video frame after the reference frame has its own corresponding actual acquisition height. To achieve image perturbation, a target acquisition height can be determined after the reference acquisition height. The target acquisition height can be lower or higher than the reference acquisition height, thus facilitating the simulation of camera movement (ascending or descending). The target acquisition height can be a height sequence or a height set, containing the target acquisition heights corresponding to each sample video frame after the reference frame.

[0154] Specific methods for determining the target acquisition height may include: To simulate a camera rising, the target acquisition height of sample video frames after the reference frame can gradually increase from the reference acquisition height, meaning the target acquisition height of each subsequent sample video frame is greater than that of the preceding sample video frame. To simulate a camera falling, the target acquisition height of sample video frames after the reference frame can gradually decrease from the reference acquisition height, meaning the target acquisition height of each subsequent sample video frame is less than or greater than that of the preceding sample video frame. To simulate a camera rising and then falling, the target acquisition height of sample video frames after the reference frame can first increase and then decrease from the reference acquisition height. To simulate a camera falling and then rising, the target acquisition height of sample video frames after the reference frame can first decrease and then increase from the reference acquisition height.

[0155] Once the target acquisition height of the sample video frames after the reference frame is determined, image transformation can be performed on the sample video frames to obtain transformed video frames. Specifically, image transformation can refer to transforming the sample video frames at the actual acquisition height into transformed video frames at the target acquisition height, that is, transforming the viewpoint of the sample video frames after the reference frame to obtain the corresponding transformed video frames.

[0156] In this embodiment, the reference frame and the transformed video frame can be directly determined as the processed sample video frame, or the processed sample video frame can be obtained by combining other image perturbation methods based on image transformation.

[0157] In the above technical solution, adjusting the acquisition height of the sample video frame can simulate the rising or falling state of the acquisition lens, which can perturb the image from the spatiotemporal sequence of image acquisition, realize the diversity of image perturbation, and improve the effect of image perturbation.

[0158] In one example, the data distribution of the training samples can also be adjusted by resampling to make the training sample data more balanced, thereby improving the training effect of the model.

[0159] In an optional embodiment, the video frame feature information may include video feature information corresponding to each of a plurality of consecutive video frames to be processed; accordingly, please refer to Figure 6 It illustrates a keyframe determination method, which may include:

[0160] S610. Input the video feature information into the target classification model, classify the video frame to be classified based on the target classification model, and obtain the target classification result of the video frame to be classified; the video frame to be classified is any video frame other than the first frame and the last frame among the multiple consecutive video frames to be processed.

[0161] S620. Based on the target classification result, determine the keyframe recognition result.

[0162] By inputting the video feature information corresponding to each of multiple consecutive video frames to be processed into the target classification model, the video frames to be classified can be classified based on the target classification model to obtain the target classification result of the video frames to be classified. Specifically, the video frame to be classified is any video frame among multiple consecutive video frames to be processed, excluding the first frame and the last frame, that is, the video frame to be classified is the middle frame among multiple consecutive video frames to be processed.

[0163] Furthermore, the position of the intermediate frame within multiple consecutive video frames to be processed can be predetermined. For example, if N (where N is an odd number) consecutive video frames to be processed are selected each time, then the (N+1) / 2th frame can be determined as the video frame to be classified. Thus, each time keyframe identification is performed, the (N+1) / 2th frame among the N (where N is an odd number) consecutive video frames to be processed is identified as a keyframe. When N is odd, in addition to determining the (N+1) / 2th frame as the video frame to be classified, the (N-1) / 2th frame can also be determined as the video frame to be classified; when N is even, N / 2 can be determined as the video frame to be classified. Therefore, the video frame to be classified can be in any other position as long as it is not the first or last frame; this embodiment does not impose specific limitations.

[0164] To identify whether a video frame to be classified is a keyframe, context is required. That is, there should be corresponding video frames before and after the video frame to be classified as reference frames, so the video frame to be classified is generally located in the middle frame position.

[0165] In the above technical solution, the video frame to be classified is the intermediate frame of multiple consecutive video frames. Thus, the video frames before and after the intermediate frame can provide a basis for classification. That is, by using the preceding and following frames as a reference for classifying the video frame to be classified, the classification accuracy can be improved.

[0166] Furthermore, for the specific classification method of the video frames to be classified, please refer to [link / reference]. Figure 7 The method may include:

[0167] S710. Based on the target classification model, perform feature analysis on the video feature information corresponding to each of the multiple consecutive video frames to be processed, and obtain feature analysis results; the feature analysis results characterize the jumping action in the video frame to be classified and the association information with the jumping action in the first video frame; the first video frame is a video frame other than the video frame to be classified among the multiple consecutive video frames to be processed.

[0168] S720. Classify the video frames to be classified based on the association information to obtain the target classification result.

[0169] The target classification model may include a feature analysis layer and a classification output layer. The feature analysis layer can perform feature analysis on the video feature information corresponding to each of multiple consecutive video frames to be processed. Specifically, it can analyze whether there is a jumping action in multiple consecutive video frames to be processed, and if there is a jumping action, the jumping height or the extension range of the jumping action. Through feature analysis, the correlation information between the jumping action in the video frame to be classified and the jumping action in the first video frame other than the video frame to be classified is obtained. The number of the first video frames is at least two.

[0170] Specifically, the association information can be a comparison between a jumping action in the video frame to be classified and a jumping action in a first video frame. In one example, the association information can be a comparison between the jump height of the jumping action in the video frame to be classified and the jump height of the jumping action in the first video frame. The comparison result can be that the jump height of the jumping action in the video frame to be classified is greater than the jump height of the jumping action in the first video frame, or the jump height of the jumping action in the video frame to be classified is less than at least one jump height of the jumping action in the first video frame. In another example, the association information can be a comparison between the extension range of the jumping action in the video frame to be classified and the extension range of the jumping action in the first video frame. The comparison result can be that the extension range of the jumping action in the video frame to be classified is greater than the extension range of the jumping action in the first video frame, or the extension range of the jumping action in the video frame to be classified is less than at least one extension range of the jumping action in the first video frame.

[0171] The classification output layer can perform classification based on the output information of the feature analysis layer to obtain the target classification result. Specifically, the target classification result can include the probability that the video frame to be classified is identified as a keyframe. Specifically, when the jump height or extension range of the jumping action in the video frame to be classified, as represented by the association information, is greater than the jump height or extension range of the jumping action in the first video frame, the probability of outputting the video frame to be classified as a keyframe is higher; otherwise, the probability of outputting the video frame to be classified as a keyframe is lower.

[0172] In the above technical solution, in jumping scenarios, feature analysis can be performed on the feature information of video frames to obtain the correlation information between jumping actions in the video frame to be classified and jumping actions in other video frames. This makes it easier to determine whether the jumping action in the video frame to be classified meets the jumping conditions, thereby further improving the accuracy of video frame classification.

[0173] In an optional embodiment, if the probability that the video frame to be classified is a keyframe in the target classification result is greater than a preset probability, that is, if the jump height of the jumping action in the video frame to be classified indicated by the target classification result is greater than the jump height of the jumping action in the first video frame, then the video frame to be classified is determined as a keyframe.

[0174] In the above technical solution, the key frame is determined by comparing the jump height of the jumping action in the video frame to be classified with the jump height of the jumping action in the first video frame. Since the jump height can be directly obtained based on the analysis of video frame feature information, the convenience and efficiency of key frame determination can be improved.

[0175] In an optional embodiment, the final target classifier can be obtained by jointly training the target classifier and the auxiliary classifier. (See [link to relevant documentation]). Figure 8It illustrates a method for training a target classifier, the training method comprising:

[0176] S810. Input the video feature information corresponding to each of the plurality of consecutive sample video frames into a first preset classifier to obtain the first classification information of the second video frame; the second video frame is any video frame other than the first frame and the last frame among the plurality of consecutive sample video frames; the first classification information represents the probability that the jumping action in the second video frame satisfies the preset jumping condition.

[0177] S820. Input the video feature information corresponding to each of the plurality of consecutive sample video frames into the second preset classifier to obtain the second classification information of the second video frame; the second classification information represents the probability that there is jump judgment auxiliary information in the second video frame.

[0178] S830. Generate first backpropagation gradient information based on the first classification information, and generate second backpropagation gradient information based on the second classification information.

[0179] S840. Based on the first backpropagation gradient information and the second backpropagation gradient information, train the first preset classifier to obtain the target classifier.

[0180] As described above in this embodiment, the target classifier may include a feature analysis layer and a classification output layer. Correspondingly, the first preset classifier may include a first feature analysis layer and a first classification output layer, and the second preset classifier may include a second feature analysis layer and a second classification output layer. The first and second feature analysis layers have the same network structure, while the first and second classification output layers have different network structures. The classification targets of the first and second preset classifiers are different, but both are associated with jumping actions, thus achieving multi-task assisted classification and recognition. For example, the classification target of the first preset classifier is the probability that a jumping action in the second video frame meets a preset jumping condition, i.e., the probability that the second video frame is identified as a keyframe. The classification target of the second preset classifier is the probability that there is jumping judgment auxiliary information in the second video frame. The existence of jumping judgment auxiliary information in the second video frame can specifically be the presence of a jumping action in the second video frame, or the scene to which the jumping action in the second video frame belongs, such as a jumping action in a photo-taking scene, a jumping action in a high jump scene, or a jumping action in a ball-playing scene. Therefore, based on different classification targets, sample video frames can correspond to different classification labels. The loss functions of the first and second preset classifiers can be the same or different.

[0181] Therefore, based on the first classification information and the first classification label corresponding to the first preset classifier, the first loss information can be determined; based on the second classification information and the second classification label corresponding to the second classifier, the second loss information can be determined. First backpropagation gradient information can be generated based on the first loss information, and second backpropagation gradient information can be generated based on the second loss information. The first and second backpropagation gradient information here both correspond to the respective feature analysis layers. That is, the model parameters of the first feature analysis layer can be updated using the first backpropagation gradient information, and the model parameters of the second feature analysis layer can be updated using the second backpropagation gradient information. Since the network structures of the first and second feature analysis layers are the same and both are used for feature analysis, when updating the backpropagation parameters of the first feature analysis layer, the first and second backpropagation gradient information can be combined to update the model parameters of the first feature analysis layer, thereby obtaining the target classifier. Specifically, the first and second backpropagation gradient information can be weighted to obtain joint gradient information, and then the first feature analysis layer can be updated based on the joint gradient information. The weight information used for weighting can be automatically determined by the first and second preset classifiers during the joint process.

[0182] In one example, the second preset classifier can be a scene classifier, which can classify jumping actions according to different jumping scenes to obtain the jumping scene to which the jumping action in the second video frame belongs; specifically, the cross-entropy loss function can be used to train the second preset classifier; the jumping scenes in this embodiment may include high jump scenes, long jump scenes, basketball scenes, badminton scenes, photography scenes, etc.

[0183] In another example, the second preset classifier can be a jump classifier, which can be used to determine whether the second video frame includes a jumping action. Specifically, multiple consecutive sample video frames can be binary classified according to whether they are in the jumping interval. For example, if they are in the jumping interval, they are classified as 1, and if they are not in the jumping interval, they are classified as 0. The second preset classifier can be trained using a binary cross-entropy loss function.

[0184] It should be noted that the second preset classifier is an auxiliary classifier, which can only be used when training the target classifier to assist in the training of the target classifier and to constrain the training conditions of the target classifier; in the actual classification, the target classifier is used directly for classification.

[0185] In the above technical solution, when training the target classifier, the model is trained based on the back gradient information of the first preset classifier and the back gradient information of the auxiliary second preset classifier to obtain the target classifier, so that the target classifier can obtain more auxiliary training information, thereby improving the classification performance of the target classifier.

[0186] In an optional embodiment, the target classifier includes multiple target sub-classifiers; see below for details. Figure 9 It demonstrates another method for training a target classifier, which includes:

[0187] S910. Input the video feature information corresponding to each of the plurality of consecutive sample video frames into a plurality of preset sub-classifiers to obtain the classification output information of the plurality of preset sub-classifiers for the third video frame; the third video frame is any video frame in the plurality of consecutive sample video frames except for the first frame and the last frame; the classification output information represents the probability that the jumping action in the third video frame satisfies the preset jumping condition.

[0188] S920. Based on the classification output information corresponding to each of the multiple preset sub-classifiers, generate the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers.

[0189] S930. Based on the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers, the multiple preset sub-classifiers are trained to obtain the multiple target sub-classifiers.

[0190] Multiple preset sub-classifiers have the same network structure, the same classification target, but different initial network parameters. Each preset sub-classifier can include a feature analysis layer and a classification output layer, and the classification model is trained through a multi-head network. The classification target of the preset sub-classifier is the probability that the jumping action in the third video frame meets the preset jumping condition, that is, the probability that the third video frame is identified as a keyframe.

[0191] Based on the classification output information and corresponding sample labels of multiple preset sub-classifiers, loss information corresponding to each preset sub-classifier is generated, thereby determining the third backpropagation gradient information corresponding to each preset sub-classifier. Similarly, the third backpropagation gradient information can be used to update the model parameters of the feature extraction layer. Thus, when updating the model parameters of the feature extraction layer of each sub-classifier, its own third backpropagation gradient information, along with the third backpropagation gradient information of other preset sub-classifiers, can be used to update the model parameters of the feature extraction layer of that preset sub-classifier. Specifically, the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers can be weighted to obtain joint gradient information, and then the feature analysis layer of the preset sub-classifier can be updated based on the joint gradient information. The weight information used for weighting can be automatically determined by the multiple preset sub-classifiers during the joint process, and the joint gradient information corresponding to each preset sub-classifier may be different. Thus, through joint training, a target classifier containing multiple target sub-classifiers can be obtained.

[0192] When a target classifier containing multiple sub-classifiers is generated during training, the video frame to be classified can be classified based on the multiple target sub-classifiers respectively to obtain the classification result corresponding to each of the multiple target sub-classifiers; information fusion is performed on the classification results corresponding to each of the multiple target sub-classifiers to obtain the target classification result.

[0193] For example, the classification result corresponding to multiple target sub-classifiers is the probability that the intermediate frame of multiple consecutive video frames to be classified is a keyframe. Therefore, the probabilities corresponding to multiple target sub-classifiers can be weighted to obtain the target probability, which can then be determined as the target classification result. When weighting the probabilities corresponding to multiple target sub-classifiers, the weights corresponding to the multiple target sub-classifiers can be the same or different. When the weights corresponding to multiple target sub-classifiers are different, the corresponding weights can be determined based on the degree of contribution of the target sub-classifier in the classification process. The greater the contribution, the greater the corresponding weight, and vice versa.

[0194] In the above technical solution, the back gradient information of multiple sub-classifiers during the training process is mutually supportive, thereby enabling multiple sub-classifiers to assist in training and improve the classification performance of multiple sub-classifiers; by integrating three target sub-classifiers, the weaker classifiers are strengthened into a strong classifier with better classification performance, thereby improving the classification performance of the target classifier; furthermore, by fusing the classification results of multiple sub-classifiers to determine the target classification result, the accuracy of the target classification result can be improved.

[0195] In one specific embodiment, by processing the target video, multiple keyframes contained in the target video can be identified. As can be seen from the above content of this embodiment, by classifying the video frames to be classified by the target classifier, the probability of the video frames to be classified being keyframes can be obtained. Therefore, if it is necessary to filter out a small number of keyframes from multiple keyframes, keyframes whose probability of being identified as keyframes is greater than or equal to the target probability can be filtered out, or a preset number of keyframes whose probability of being identified as keyframes is relatively high can be filtered out.

[0196] In the scenario of selecting a video cover, if there are multiple keyframes identified from the target video, the keyframe with the highest probability can be selected as the video cover.

[0197] In the scene of editing highlights, if it is necessary to select a preset number of highlights from the target video, the preset number of keyframes that are more likely to be identified as keyframes can be filtered out, and then the corresponding highlights can be generated by combining the frames before and after the filtered keyframes.

[0198] In a specific example, please refer to Figure 10 The diagram illustrates a keyframe recognition model, which may include a feature extraction model and a target classifier. In this embodiment, the feature extraction model and the target classifier can be trained separately, i.e., the feature extraction model and the target classifier are pre-trained, and then the keyframe recognition model is formed based on the pre-trained feature extraction model and the target classifier. In this embodiment, the feature extraction model and the target classifier can be jointly trained, i.e., the keyframe recognition model is trained as a whole. The overall training details are similar to the pre-training details and will not be repeated here.

[0199] Taking the acquisition of 5 consecutive video frames for processing as an example, image enhancement processing is performed on the 5 consecutive video frames to obtain enhanced video frames. The enhanced video frames are then cropped, scaled, and normalized to obtain normalized video frames. The normalized video frames are then input into a feature extraction model with temporal modeling capabilities for feature extraction to obtain video frame feature information. This video frame feature information is then input into a target classifier for classification to obtain the probability that the 3rd frame is a keyframe. The target classifier may include one or more classifiers, and the multiple classifiers have the same structure.

[0200] As can be seen from the technical solutions provided in the embodiments of this specification above, in the embodiments of this specification, by acquiring a small number of continuous video frames to be processed each time and a lightweight deep learning model, real-time recognition of key frames can be achieved; and by training the classifier based on the multi-task training method and the multi-head classification training method, the classification performance of the classifier can be improved.

[0201] It should be noted that the methods described above in this embodiment can be combined according to specific implementation conditions and have corresponding beneficial effects, which will not be elaborated here.

[0202] Figure 11 A video processing apparatus according to an exemplary embodiment includes:

[0203] The first acquisition module 1110 is used to acquire multiple consecutive video frames to be processed from the target video; the number of the multiple consecutive video frames to be processed is less than the number of video frames contained in the target video.

[0204] The feature extraction module 1120 is used to extract features from the multiple consecutive video frames to be processed to obtain video frame feature information;

[0205] The keyframe recognition module 1130 is used to perform real-time keyframe recognition based on the video frame feature information, and to determine the keyframe recognition results corresponding to the multiple consecutive video frames to be processed.

[0206] In an optional embodiment, the feature extraction module 1120 includes:

[0207] The feature information determination module is used to extract features from the multiple consecutive video frames to be processed based on the feature extraction model to obtain the feature information of the video frames;

[0208] The device further includes:

[0209] The second acquisition module is used to acquire processed sample video frames; the processed sample video frames are obtained by performing image perturbation processing on the original sample video frames.

[0210] The first training module is used to train a preset machine learning model based on the processed sample video frames to obtain the feature extraction model.

[0211] In an optional embodiment, the apparatus further includes:

[0212] The pixel perturbation module is used to perform pixel perturbation processing on each pixel of the original sample video frame to obtain a pixel-perturbated video frame.

[0213] The first generation module is used to generate the processed sample video frame based on the pixel-perturbed video frame.

[0214] In an optional embodiment, the original sample video frames comprise a plurality of consecutive sample video frames; the apparatus further comprises:

[0215] The first determining module is used to determine the first frame of the plurality of consecutive sample video frames as the reference frame;

[0216] The second determining module is used to determine the reference acquisition height of the reference frame; the reference acquisition height represents the acquisition height of the image acquisition device when acquiring the reference frame;

[0217] The acquisition height determination module is used to determine the target acquisition height of sample video frames after the reference frame based on the reference acquisition height.

[0218] The image transformation module is used to perform image transformation on the sample video frames after the reference frame based on the target acquisition height to obtain transformed video frames;

[0219] The second generation module is used to generate the processed sample video frame based on the reference frame and the transformed video frame.

[0220] In an optional embodiment, the keyframe recognition module 1130 includes:

[0221] The first classification module is used to input the video feature information into the target classification model, classify the video frame to be classified based on the target classification model, and obtain the target classification result of the video frame to be classified; the video frame to be classified is any video frame other than the first frame and the last frame among the multiple consecutive video frames to be processed.

[0222] The third determining module is used to determine the keyframe recognition result based on the target classification result.

[0223] In an optional embodiment, the first classification module includes:

[0224] The feature analysis module is used to perform feature analysis on the video feature information corresponding to each of the multiple consecutive video frames to be processed based on the target classification model, and obtain feature analysis results; the feature analysis results characterize the jumping action in the video frame to be classified and the association information with the jumping action in the first video frame; the first video frame is a video frame other than the video frame to be classified among the multiple consecutive video frames to be processed.

[0225] The second classification module is used to classify the video frames to be classified based on the association information to obtain the target classification result.

[0226] In an optional embodiment, the third determining module includes:

[0227] The fourth determining module is used to determine the video frame to be classified as the keyframe when the jump height of the jumping action in the video frame to be classified, as indicated by the target classification result, is greater than the jump height of the jumping action in the first video frame.

[0228] In an optional embodiment, the device includes:

[0229] The first input module is used to input the video feature information corresponding to each of the plurality of consecutive sample video frames into the first preset classifier to obtain the first classification information of the second video frame; the second video frame is any video frame other than the first frame and the last frame among the plurality of consecutive sample video frames; the first classification information represents the probability that the jumping action in the second video frame satisfies the preset jumping condition.

[0230] The second input module is used to input the video feature information corresponding to each of the multiple consecutive sample video frames into the second preset classifier to obtain the second classification information of the second video frame; the second classification information represents the probability that there is jump judgment auxiliary information in the second video frame.

[0231] The third generation module is used to generate first backpropagation gradient information based on the first classification information and to generate second backpropagation gradient information based on the second classification information.

[0232] The second training module is used to train the first preset classifier based on the first backpropagation gradient information and the second backpropagation gradient information to obtain the target classifier.

[0233] In an optional embodiment, the target classifier includes multiple target sub-classifiers, and the apparatus further includes:

[0234] The third input module is used to input the video feature information corresponding to each of the multiple consecutive sample video frames into multiple preset sub-classifiers to obtain the classification output information of the multiple preset sub-classifiers for the third video frame; the third video frame is any video frame in the multiple consecutive sample video frames except for the first frame and the last frame; the classification output information represents the probability that the jumping action in the third video frame satisfies the preset jumping condition.

[0235] The fourth generation module is used to generate the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers based on the classification output information corresponding to each of the multiple preset sub-classifiers.

[0236] The third training module is used to train the multiple preset sub-classifiers based on the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers, so as to obtain the multiple target sub-classifiers.

[0237] The first classification module includes:

[0238] The third classification module is used to classify the video frame to be classified based on the multiple target sub-classifiers respectively, and obtain the classification results corresponding to each of the multiple target sub-classifiers.

[0239] The information fusion module is used to fuse the classification results of the multiple target sub-classifiers to obtain the target classification result.

[0240] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0241] Figure 12 This is a block diagram illustrating an electronic device for video processing according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 12 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a video processing method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0242] Figure 13 This is a block diagram illustrating an electronic device for video processing according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 13 As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a video processing method.

[0243] Those skilled in the art will understand that Figure 12 and Figure 13The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0244] In an exemplary embodiment, a storage medium is also provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the methods described in the embodiments of this disclosure.

[0245] In an exemplary embodiment, a computer program product including instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods in the embodiments of this disclosure.

[0246] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0247] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0248] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that, include: Multiple consecutive video frames to be processed are obtained from the target video; the number of the multiple consecutive video frames to be processed is less than the number of video frames contained in the target video. Feature extraction is performed on the multiple consecutive video frames to be processed to obtain video frame feature information; Real-time keyframe recognition is performed based on the video frame feature information to determine the keyframe recognition results corresponding to the multiple consecutive video frames to be processed; the real-time keyframe recognition based on the video frame feature information to determine the keyframe recognition results corresponding to the multiple consecutive video frames to be processed includes: The video frame feature information is input into the target classification model, and the video frame to be classified is classified based on the target classification model to obtain the target classification result of the video frame to be classified; the video frame to be classified is any video frame other than the first frame and the last frame among the multiple consecutive video frames to be processed. Based on the target classification result, the keyframe recognition result is determined; the determination of the keyframe recognition result based on the target classification result includes: If the jump height of the jumping action in the video frame to be classified, as indicated by the target classification result, is greater than the jump height of the jumping action in the first video frame, the video frame to be classified is determined as a keyframe; the first video frame is a video frame other than the video frame to be classified among the plurality of consecutive video frames to be processed.

2. The method according to claim 1, characterized in that, The step of extracting features from the multiple consecutive video frames to be processed to obtain video frame feature information includes: Based on the feature extraction model, feature extraction is performed on the multiple consecutive video frames to be processed to obtain the feature information of the video frames; The method further includes a method for training the feature extraction model, the training method of the feature extraction model including: Obtain processed sample video frames; the processed sample video frames are obtained by performing image perturbation processing on the original sample video frames. The preset machine learning model is trained based on the processed sample video frames to obtain the feature extraction model.

3. The method according to claim 2, characterized in that, The method includes an image perturbation processing method, which comprises: Pixel perturbation processing is performed on each pixel of the original sample video frame to obtain a pixel-perturbed video frame; The processed sample video frame is generated based on the pixel-perturbed video frame.

4. The method according to claim 2, characterized in that, The original sample video frames include multiple consecutive sample video frames; the method includes an image perturbation processing method, which includes: The first frame of the plurality of consecutive sample video frames is determined as the reference frame; Determine the reference acquisition height of the reference frame; the reference acquisition height represents the acquisition height of the image acquisition device when acquiring the reference frame; Based on the reference acquisition height, the target acquisition height of the sample video frames after the reference frame is determined; Based on the target acquisition height, image transformation is performed on the sample video frames after the reference frame to obtain transformed video frames; The processed sample video frame is generated based on the reference frame and the transformed video frame.

5. The method according to claim 1, characterized in that, The process of classifying the video frames to be classified based on the target classification model to obtain the target classification result for the video frames to be classified includes: Based on the target classification model, feature analysis is performed on the video frame feature information corresponding to each of the multiple consecutive video frames to be processed to obtain feature analysis results; the feature analysis results characterize the jumping action in the video frame to be classified and the correlation information between the jumping action in the first video frame. The video frames to be classified are classified based on the association information to obtain the target classification result.

6. The method according to claim 4, characterized in that, The method includes a method for training the target classification model, the training method comprising: The video frame feature information corresponding to each of the multiple consecutive sample video frames is input into a first preset classifier to obtain the first classification information for the second video frame; the second video frame is any video frame among the multiple consecutive sample video frames except for the first frame and the last frame; the first classification information represents the probability that the jumping action in the second video frame satisfies the preset jumping condition. The video frame feature information corresponding to each of the multiple consecutive sample video frames is input into the second preset classifier to obtain the second classification information of the second video frame; the second classification information represents the probability that there is jump judgment auxiliary information in the second video frame. First backpropagation gradient information is generated based on the first classification information, and second backpropagation gradient information is generated based on the second classification information; Based on the first backpropagation gradient information and the second backpropagation gradient information, the first preset classifier is trained to obtain the target classification model.

7. The method according to claim 4, characterized in that, The target classification model includes multiple target sub-classifiers, and the method further includes a training method for the multiple target sub-classifiers, the training method including: The video frame feature information corresponding to each of the multiple consecutive sample video frames is input into multiple preset sub-classifiers to obtain the classification output information of the multiple preset sub-classifiers for the third video frame; the third video frame is any video frame in the multiple consecutive sample video frames except for the first frame and the last frame; the classification output information represents the probability that the jumping action in the third video frame satisfies the preset jumping condition. Based on the classification output information corresponding to each of the multiple preset sub-classifiers, the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers is generated. Based on the third backpropagation gradient information corresponding to each of the multiple preset sub-classifiers, the multiple preset sub-classifiers are trained to obtain the multiple target sub-classifiers. The process of classifying the video frames to be classified based on the target classification model to obtain the target classification result for the video frames to be classified includes: The video frames to be classified are classified based on the multiple target sub-classifiers respectively, and the classification results corresponding to each of the multiple target sub-classifiers are obtained. The classification results of the multiple target sub-classifiers are fused to obtain the target classification result.

8. A video processing apparatus, comprising: The first acquisition module is used to acquire multiple consecutive video frames to be processed from the target video; the number of the multiple consecutive video frames to be processed is less than the number of video frames contained in the target video. The feature extraction module is used to extract features from the multiple consecutive video frames to be processed to obtain video frame feature information; A keyframe recognition module is used to perform real-time keyframe recognition based on the video frame feature information, and determine the keyframe recognition results corresponding to the plurality of consecutive video frames to be processed; the real-time keyframe recognition based on the video frame feature information to determine the keyframe recognition results corresponding to the plurality of consecutive video frames to be processed includes: inputting the video frame feature information into a target classification model, classifying the video frames to be classified based on the target classification model, and obtaining a target classification result for the video frames to be classified; the video frames to be classified are any video frames among the plurality of consecutive video frames to be processed except for the first frame and the last frame; determining the keyframe recognition result based on the target classification result; the determination of the keyframe recognition result based on the target classification result includes: if the target classification result indicates that the jump height of the jump action in the video frame to be classified is greater than the jump height of the jump action in the first video frame, the video frame to be classified is determined as a keyframe; the first video frame is a video frame among the plurality of consecutive video frames to be processed except for the video frame to be classified.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the video processing method as described in any one of claims 1 to 7.

Images