Video processing method and device, equipment and medium

Abstract

The embodiment of the invention provides a video processing method and device, equipment and a medium. The method comprises the steps of determining a scene type corresponding to a to-be-processed video, and determining a target feature extraction strategy of the to-be-processed video based on the scene type; determining a motion feature corresponding to the to-be-processed video based on the target feature extraction strategy, and determining the motion intensity of the to-be-processed video based on the motion feature; determining a filtering parameter of the to-be-processed video based on the motion intensity, and filtering the to-be-processed video based on the filtering parameter to obtain filtered video data; and determining a spatial transformation parameter corresponding to the to-be-processed video based on the filtered video data, and performing inter-frame transformation on the to-be-processed video based on the spatial transformation parameter to obtain a stable video sequence. The method is used for achieving the effect of effectively improving the stability and the visual quality of the video in a complex dynamic scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and in particular to a video processing method, device, equipment and medium. BACKGROUND

[0002] With the popularity of mobile intelligent terminals and portable shooting devices, video collection in dynamic scenes such as handheld shooting, unmanned aerial vehicle aerial photography, and vehicle-mounted recorders has become routine. However, in these scenes, due to the non-stationary motion state of the shooting device itself (for example, physiological shaking of the operator's hand, attitude change of the unmanned aerial vehicle affected by the airflow, jolt in the vehicle driving, etc.), the collected original video sequence inevitably introduces severe and unexpected inter-frame jitter. Such jitter not only seriously damages the visual perception of the video, causing the viewer to be dizzy or tired, but also interferes with subsequent computer vision analysis tasks based on the video (such as target tracking, three-dimensional reconstruction, motion recognition, etc.), reducing its accuracy and reliability.

[0003] In related technologies, video stabilization technology generally uses fixed feature extraction and motion filtering strategies, which cannot be adjusted according to the dynamic complexity of the video scene, resulting in poor video processing effect and affecting the user's experience. SUMMARY

[0004] The embodiments of the present application provide a video processing method, device, equipment and medium, and a computer program product, to effectively improve the stability and visual quality of the video in complex dynamic scenes.

[0005] In a first aspect, the embodiments of the present application provide a video processing method, comprising: determining a scene type corresponding to a to-be-processed video, determining a target feature extraction strategy of the to-be-processed video based on the scene type; determining a motion feature corresponding to the to-be-processed video based on the target feature extraction strategy, and determining a motion intensity of the to-be-processed video based on the motion feature; determining a filtering parameter of the to-be-processed video based on the motion intensity, and performing filtering processing on the to-be-processed video based on the filtering parameter to obtain filtered video data; determining a spatial transformation parameter corresponding to the to-be-processed video based on the filtered video data, and performing inter-frame transformation on the to-be-processed video based on the spatial transformation parameter to obtain a stable video sequence.

[0006] In a possible implementation, the method further includes: extracting a plurality of video frames of the to-be-processed video based on the target feature extraction strategy, the plurality of video frames being adjacent video frames in the to-be-processed video; determining a motion feature corresponding to the to-be-processed video based on the plurality of video frames, and determining a motion intensity of the to-be-processed video based on the motion feature; and determining a filter parameter of a target filter based on the motion intensity, the filter parameter including a process noise covariance and a measurement noise covariance.

[0007] In a possible implementation, the determining the motion feature corresponding to the to-be-processed video based on the plurality of video frames, and the determining the motion intensity of the to-be-processed video based on the motion feature, includes: determining a motion parameter corresponding to the plurality of video frames, and determining a corresponding motion feature based on the motion parameter, the motion feature including a mean value and a covariance; and determining the motion intensity of the to-be-processed video based on the mean value and the covariance.

[0008] In a possible implementation, the determining the scene type corresponding to the to-be-processed video includes: determining an image feature statistical quantity variance corresponding to the to-be-processed video, the image feature statistical quantity variance including a gradient entropy and a brightness variance; and determining the scene type corresponding to the to-be-processed video based on the gradient entropy and the brightness variance.

[0009] In a possible implementation, the determining the motion feature corresponding to the to-be-processed video based on the target feature extraction strategy includes: determining a down-sampling ratio of the to-be-processed video based on the scene type, and performing a sampling process on the to-be-processed video based on the down-sampling ratio to obtain sampled video data; and performing feature extraction on the sampled video data based on the target feature extraction strategy to determine the motion feature corresponding to the to-be-processed video.

[0010] In a possible implementation, the method further includes: obtaining a difference between a current frame rate and a target frame rate of the to-be-processed video, and a real-time load parameter of a video processing device; and adjusting the down-sampling ratio based on the difference and the real-time load parameter, to perform a sampling process on the to-be-processed video based on the adjusted down-sampling ratio.

[0011] In a possible implementation, the determining the spatial transformation parameter corresponding to the to-be-processed video based on the filtered video data comprises: determining a motion amplitude of the to-be-processed video based on the filtered video data, and determining a cropping boundary parameter of the to-be-processed video based on the motion amplitude; determining a motion smoothing parameter of the to-be-processed video based on the filtered video data, and determining a geometric transformation parameter of the to-be-processed video based on the motion smoothing parameter; and determining the spatial transformation parameter of the to-be-processed video based on the cropping boundary parameter and the geometric transformation parameter.

[0012] In a second aspect, an embodiment of the present application provides a video processing apparatus, which comprises: a determining module configured to determine a scene type corresponding to a to-be-processed video, and determine a target feature extraction strategy of the to-be-processed video based on the scene type; a feature extraction module configured to determine a motion feature corresponding to the to-be-processed video based on the target feature extraction strategy, and determine a motion intensity of the to-be-processed video based on the motion feature; a filtering module configured to determine a filtering parameter of the to-be-processed video based on the motion intensity, and perform filtering processing on the to-be-processed video based on the filtering parameter to obtain filtered video data; and a transformation module configured to determine a spatial transformation parameter corresponding to the to-be-processed video based on the filtered video data, and perform inter-frame transformation on the to-be-processed video based on the spatial transformation parameter to obtain a stable video sequence.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which comprises: a memory and a processor; the memory stores computer-executed instructions; and the processor executes the computer-executed instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executed instructions, and the computer-executed instructions are executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0016] The video processing method, device, equipment and medium provided by the embodiments of the present application can select a suitable target feature extraction strategy by determining the scene type of the video to be processed, can make the feature extraction process more targeted, and can more accurately capture key feature information. Then, the motion intensity is determined based on the extracted features, which can provide a reliable basis for the determination of subsequent filtering parameters, ensures that the filtering process can effectively remove noise and other interference factors, and will not be excessively smoothed to cause video information loss, determines the spatial transformation parameter according to the filtered data and performs inter-frame transformation, so that the finally obtained spatial transformation parameter is more consistent with the actual motion of the video, thereby the video can be more effectively stabilized, a video sequence with higher quality and stability is obtained, and the effect and efficiency of video stabilization are significantly improved, and the stability and visual quality of the video in a complex dynamic scene are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] Figure 1 A scene diagram of video processing provided by the present application is provided.

[0019] Figure 2 A flowchart of the video processing method provided by the present application is provided. Figure 1 ;

[0020] Figure 3 A flowchart of the video processing method provided by the present application is provided. Figure 2 ;

[0021] Figure 4 A structural diagram of the video processing device provided by the present application is provided.

[0022] Figure 5 A structural diagram of the electronic device provided by the present application is provided.

[0023] Through the above drawings, the specific embodiments of the present application have been shown, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0024] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description herein relates to the drawings, where like numbers represent similar or corresponding elements throughout the several views. The following detailed description is not representative of all embodiments consistent with the present application. Instead, it is merely exemplary of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] Firstly, the terms involved in the present application are explained:

[0026] Down-sampling is a common operation in the field of signal processing or data processing, which is to thin out data according to certain rules and sampling rates. For video or image data, it is to selectively discard part of the original pixel points or data points to reduce the resolution or sampling density of the data. For example, in video processing, the size and data volume of the video can be reduced by reducing the number of pixels in the video frame, which can reduce the computational burden of subsequent processing to some extent, but may also cause the loss of some detailed information, so it is necessary to reasonably set the down-sampling parameters according to the actual needs.

[0027] Kalman filter is a kind of efficient and widely used recursive filtering algorithm, which combines prediction model and observation data to make optimal estimation of target state in dynamic system with noise. The algorithm uses system state equation to describe the prediction process of state, and then fuses the predicted value with the actual observation value through observation equation, dynamically adjusts the weight according to the uncertainty (measured by covariance matrix) of the two, and continuously corrects the estimation result, which plays a key role in navigation, target tracking, signal processing and many other fields, can effectively remove noise interference, and provide accurate state estimation.

[0028] Please refer to Figure 1 , Figure 1 The application scenario diagram of video processing provided for an exemplary embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the video processing system 100 is composed of a video source 101, a video encoder 102, a video decoder 103 and a video sink 104. Figure 1As shown, the application scenario includes the intelligent terminal 110 and the server end 120, the intelligent terminal 110 is used to collect the video to be processed, specifically, the server end 120 and the video to be processed, through the server end 120, the scene type corresponding to the video to be processed is determined, and further, based on the scene type, the target feature extraction strategy of the video to be processed is determined, and then the motion feature corresponding to the video to be processed can be determined according to the target feature extraction strategy, so as to determine the motion intensity of the video to be processed based on the motion feature. Then, the server end 120 determines the filtering parameter of the video to be processed based on the motion intensity, and performs filtering processing on the video to be processed based on the filtering parameter, so as to obtain the filtered video data. Finally, based on the filtered video data, the spatial transformation parameter corresponding to the video to be processed is determined, and the interframe transformation of the video to be processed is performed based on the spatial transformation parameter, so that the stable video sequence is output on the intelligent terminal 110.

[0029] wherein, Figure 1 The intelligent terminal 110 as shown can be a smart phone, a vehicle-mounted computer, a tablet computer, a drone, a notebook computer or a wearable device, etc. any terminal device supporting video collection, but is not limited thereto. Figure 1 The server end 120 as shown can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, etc. and is not limited herein. The intelligent terminal 110 can communicate with the server end 120 through 3G (third generation mobile information technology), 4G (fourth generation mobile information technology), 5G (fifth generation mobile information technology) and other wireless networks, and is not limited herein.

[0030] In the field of video acquisition, with the popularization and diversification of shooting devices, such as handheld shooting devices and aerial shooting devices like drones, the flexibility and convenience of video acquisition have been greatly improved. However, in these practical application scenarios, it is often difficult for shooting devices to maintain an absolutely stable motion state. For example, when shooting by hand, physiological tremors of the hand will inevitably be transmitted to the shooting device; during aerial photography by drone, factors such as wind force and flight attitude adjustment will cause the drone to change its attitude, and thus the shooting device mounted thereon will exhibit non-stationary motion. This non-stationary motion makes the video sequence collected prone to inter-frame jitter phenomenon. Inter-frame jitter not only seriously damages the visual coherence of the video and reduces the visual experience of the audience, but also greatly interferes with subsequent video-based analysis and processing work, such as target tracking, behavior recognition, three-dimensional reconstruction, etc., leading to inaccurate or even incorrect analysis results.

[0031] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described again in some examples. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0032] Figure 2 Flowchart of the video processing method provided by the present application Figure 1 As shown in Figure 2 The flowchart of the video processing method at least includes steps S201 to S204, which are described in detail as follows:

[0033] Step S201, determine the scene type corresponding to the video to be processed, and determine the target feature extraction strategy of the video to be processed based on the scene type.

[0034] Exemplarily, the input video to be processed is preliminarily analyzed, and multiple basic feature information is extracted from each frame of the video, which includes gradient information of the image. The gradient amplitude and direction are obtained by calculating the gray level change rate of the pixel points in different directions of the image, and then the texture structure complexity of the image is analyzed. Because scenes with different texture complexity have obvious differences in gradient distribution, for example, the gradient of a scene with rich texture changes frequently and has various amplitudes, while the gradient of a low-texture scene changes relatively gently and is concentrated; at the same time, the brightness information of the image is extracted, and statistical quantities such as brightness mean and variance of the image as a whole are calculated. The brightness mean can reflect the overall brightness of the scene, and the brightness variance can reflect the severity of the light change. The brightness variance value of a scene with large light change will be larger.

[0035] After obtaining these basic feature information, a pre-trained scene classification model is used, which is trained based on a large number of annotated scene type video data and has the ability to identify scene types from input feature information. Specifically, the extracted gradient and brightness features are input into the model, and the model outputs the scene type corresponding to the video to be processed after complex calculation and judgment, which may include low texture scene, high light change scene, normal texture and stable light scene and other types. After determining the scene type, the determination of the target feature extraction strategy is entered, and different scene types have adaptive feature extraction algorithms and parameter settings. For low texture scenes, the conventional feature point extraction algorithm may not be able to obtain enough effective feature points, so the algorithm with less texture dependence and the ability to capture local significant structure is selected, such as the oriented fast feature point detection and rotated binary robust independent element feature descriptor algorithm (Oriented FAST and Rotated BRIEF, ORB), which combines key point detection and descriptor, has the characteristics of fast and certain adaptability to low texture, and appropriately adjusts the threshold parameter of key point detection to increase the number of key points detected in low texture area. For high light change scenes, light changes will cause unstable feature description, so an algorithm with light invariance is selected, such as the scale-invariant feature transform algorithm (Scale-Invariant Feature Transform, SIFT), which detects key points by constructing Gaussian pyramid and (Difference of Gaussian, DOG) DOG pyramid, and uses gradient histogram to generate descriptor, which has good robustness to light change. However, considering its large amount of calculation, some parameters can be optimized to improve processing speed while ensuring feature extraction quality. For normal texture and stable light scenes, a good feature tracking algorithm (Good Features to Track, GFTT) with good comprehensive performance and high computational efficiency can be used, which detects key points based on the corner response function of the image, and can ensure the quality of feature points while having fast processing speed.

[0036] Step S202, determine the motion feature corresponding to the video to be processed based on the target feature extraction strategy, and determine the motion intensity of the video to be processed based on the motion feature.

[0037] For example, following a defined target feature extraction strategy, corresponding feature points are extracted from each frame of the video to be processed. For instance, if the ORB algorithm is used, its speed and adaptability to low-texture images are utilized to detect corner points with significant local structures as feature points in each frame. If the SIFT algorithm is used, its Gaussian and Dog pyramids are used to accurately locate key points that remain stable under varying lighting conditions as feature points. When using the GFTT algorithm, suitable feature points are selected based on the corner response function of the image. After successfully extracting the feature points of each frame, optical flow or feature matching algorithms are used to analyze the motion of feature points between adjacent frames. Optical flow calculates the instantaneous velocity of pixels in the image to obtain the motion vector of feature points between frames. It is based on the relationship between the temporal change of image grayscale and the motion of objects in the scene, assuming that the image grayscale remains constant, and obtains the motion vector by solving the optical flow constraint equation. Feature matching algorithms, on the other hand, match feature points in the current frame with feature points in the next frame to find identical or similar feature point pairs, and then determine the displacement changes between them to obtain the motion information of the feature points.

[0038] After obtaining the motion information of feature points between adjacent frames using the above method, this motion information is integrated and analyzed to determine the motion characteristics of the video to be processed. Motion characteristics can encompass the directional distribution of motion, i.e., statistically analyzing the directions of motion vectors of all feature points. Analyzing the main directional trends of object motion in the video also includes the amplitude distribution of motion, calculating the magnitude of the motion vector of each feature point to determine the intensity of motion at different feature points. After obtaining the motion characteristics, the motion intensity of the video to be processed is further determined by calculating the average magnitude of the motion vectors of all feature points. This average value can reflect the overall average motion intensity of the video to a certain extent; the larger the average value, the more intense the overall motion of the video, and the higher the motion intensity.

[0039] In addition, in some feasible implementations, the concentration of motion direction can also be considered. If the motion direction of most feature points is relatively concentrated, it indicates that the video has relatively consistent large-amplitude motion and relatively large motion intensity. Conversely, if the motion direction of feature points is dispersed and the motion amplitude is generally small, it indicates that the motion intensity of the video is low.

[0040] Step S203: Determine the filtering parameters of the video to be processed based on the motion intensity, and perform filtering processing on the video to be processed based on the filtering parameters to obtain filtered video data.

[0041] Exemplarily, after the motion intensity of the video to be processed is obtained, a key stage of determining the filtering parameter according to the motion intensity and performing filtering processing to obtain the filtered video data is entered. Among them, the motion intensity is an important indicator for measuring the overall motion intensity of the video, and the numerical value can directly reflect the amplitude and frequency of the motion of the objects in the video. Therefore, based on this, the filtering parameter is determined, which can enable the filtering processing to more targetedly eliminate the noise and interference caused by the unexpected motion such as device jitter in the video, while retaining the effective information in the video as much as possible. Specifically, when the motion intensity is large, it means that there is relatively intense jitter or rapid motion change in the video, and relatively strong filtering processing is needed to suppress the adverse effects caused by the intense motion. When determining the filtering parameter, the cutoff frequency of the filtering or the kernel size of the filter and other parameters are increased. For example, for a Gaussian filter, the standard deviation is increased, so that the filter can filter out more high-frequency jitter components in the frequency domain, and the influence range of the filtering is expanded in the spatial domain, so that the intense jitter in the video is more effectively smoothed, and the video picture looks more stable and smooth. On the contrary, when the motion intensity is small, it means that the motion in the video is relatively flat, and the jitter is not obvious, and if too strong filtering processing is used, some detailed information in the video may be lost due to excessive smoothing, affecting the quality of the video. Therefore, relatively weak filtering parameters are selected, such as reducing the standard deviation of the Gaussian filter to reduce the strength of the filtering, while ensuring that the noise caused by the slight jitter can be removed, and the details and edge information in the video are retained to the greatest extent.

[0042] After the appropriate filtering parameter is determined, the corresponding filtering algorithm is used to perform filtering processing on the entire video to be processed. The filtering algorithm will operate on each frame of the video according to the determined parameter. Taking the Gaussian filter as an example, it will perform weighted average calculation on each pixel point and the pixel points in the neighborhood of the pixel point, and the weight is determined according to the Gaussian function. The closer the pixel point is to the center pixel point, the greater the weight. Through such a calculation method, the noise and jitter in the image can be effectively smoothed, while the overall structure and characteristics of the image are maintained. After the filtering processing, the unexpected jitter and noise in the video are significantly suppressed, and the filtered video data is obtained.

[0043] In step S204, the spatial transformation parameter corresponding to the video to be processed is determined based on the filtered video data, and the inter-frame transformation of the video to be processed is performed based on the spatial transformation parameter to obtain a stable video sequence.

[0044] Exemplarily, key feature points are extracted from the filtered video frames, which need to have stability and traceability, such as corner points, edge points, etc. which are regularly changed in position in consecutive frames and are not easy to lose. Further, the feature points of adjacent frames are associated and matched by a feature matching algorithm to construct a feature point pair set, and then the relative change relationship of the feature point pairs in the spatial position is analyzed to deduce the spatial transformation model between frames, such as affine transformation model, perspective transformation model, etc. The parameters in the model are spatial transformation parameters, which can quantitatively describe the translation, rotation, scaling and other motion transformation conditions between frames.

[0045] Optionally, in the process of determining the spatial transformation parameters, the continuity and consistency of the transformation can be considered comprehensively, the transformation parameter change trend between multiple adjacent frames is counted, the abnormal values are eliminated, the parameters are smoothed and optimized to ensure the accuracy and stability of the transformation parameters and to avoid poor overall transformation effect caused by errors of individual frames. Further, after obtaining the optimized spatial transformation parameters, inter-frame transformation operation is performed on each frame of video according to the parameters, specifically, for the current frame, according to the spatial transformation parameters of the current frame and the reference frame (such as the first frame or the previous frame of the video), the coordinate transformation is performed on each pixel point in the current frame to adjust the pixel point to the vicinity of the corresponding position in the reference frame, eliminate the position offset between frames caused by device movement, and make each frame keep aligned in spatial position with the reference frame. After such inter-frame transformation processing, each frame in the video sequence realizes accurate spatial position correction, the inter-frame misplacement caused by shaking is effectively corrected, and finally a stable video sequence with visual coherence and smooth motion is obtained.

[0046] Optionally, stable feature points such as corner points or edge points are extracted from the filtered video frames, which need to maintain positional coherence in consecutive frames to support accurate matching. Feature matching algorithms such as SIFT or ORB are used to associate feature point pairs in adjacent frames, and after constructing a matching set, a spatial transformation model (such as affine transformation or homography) is fitted using a related robust estimation method. The translation, rotation, scaling and other parameters in the model constitute the geometric transformation parameters, which are used to quantify the motion offset between frames. At the same time, since geometric transformation may cause some pixels to move out of the picture boundary or introduce blank areas, the cropped area needs to be calculated in combination with the boundary information of the transformed image. Specifically, by analyzing the influence of the transformation matrix on the coordinates of the pixels at the four corners of the image, the minimum bounding rectangle of the effective picture area is determined, and the boundary coordinates are the cropping parameters, which ensure that the transformed picture has no black edges and retains the core content. In the parameter optimization stage, time series smoothing processing (such as moving average filtering) is required for multiple frames of transformation parameters to eliminate parameter fluctuations caused by single-frame matching errors and improve transformation stability. Finally, according to the optimized geometric transformation parameters, each frame is mapped. For example, the pixel points are converted from the current frame coordinate system to the reference frame coordinate system to achieve inter-frame position alignment, and then the boundary cropping parameters are applied to cut off the invalid edge area to ensure that each frame of picture strictly matches the reference frame in spatial position and visual range. The final output stable video sequence eliminates the frame-to-frame misalignment caused by device jitter, the picture is coherent and smooth, and the edge processing is natural.

[0047] In the embodiments provided in the present application, by customizing the feature extraction strategy according to the video scene type and analyzing the motion features and intensity accordingly, the filter parameters are adaptively adjusted for targeted denoising, and finally the spatial transformation parameters are used to complete the inter-frame image stabilization processing, which can effectively improve the stability and visual quality of the video in complex dynamic scenes.

[0048] Based on the above embodiments, in one of the example embodiments provided in the present application, the specific implementation process of the above-mentioned video processing method can further include steps S301 to S303, which are described in detail as follows:

[0049] Step S301, extracting a plurality of video frames of a to-be-processed video based on a target feature extraction strategy, the plurality of video frames being adjacent video frames in the to-be-processed video.

[0050] Step S302, determining a motion feature corresponding to the to-be-processed video based on the plurality of video frames, and determining a motion intensity of the to-be-processed video based on the motion feature.

[0051] Step S303, determining a filter parameter of a target filter based on the motion intensity, the filter parameter including a process noise covariance and a measurement noise covariance.

[0052] For example, the scene type determined according to the preceding steps (such as a low-texture, high-illumination variation, etc.) dynamically selects a suitable feature extraction algorithm (such as a GFTT algorithm, an ORB algorithm, or a SIFT algorithm). For example, in a low-texture scene, an ORB algorithm with less texture dependence is selected, and in a high-illumination variation scene, a SIFT algorithm with illumination invariance is selected. Stable and traceable feature points (such as corner points, edge points) are extracted from adjacent video frames by using these algorithms, and a feature point pair set is formed by using a feature matching algorithm to establish a corresponding relationship between feature points in adjacent frames. Subsequently, the inter-frame motion feature can be analyzed based on these feature point pairs, the direction distribution (such as the trend of the main motion direction) and the amplitude distribution (such as the change in the modulus of the motion vector) of the motion are calculated by calculating the displacement vector of the feature point pair, and then the motion intensity of the video to be processed is quantified. Specifically, the average value or variance of the modulus of the motion vector of all feature points can be used to measure the degree of overall motion.

[0053] Optionally, the filtering parameters of the target filter (such as a Kalman filter) can also be adjusted according to the quantification result of the motion intensity, wherein the process noise covariance reflects the uncertainty of the change of the system state (such as the motion trajectory) over time, and when the motion intensity is large, such as in a scene with violent shaking, the process noise covariance can be appropriately increased to allow the state estimation to more flexibly track the rapidly changing motion. The measurement noise covariance reflects the observation noise level of the feature matching or motion estimation, and can be dynamically adjusted according to the reliability of the feature point matching, for example, the number of matching point pairs, the matching error, etc. Specifically, when the feature matching accuracy is high, the measurement noise covariance is reduced to increase the weight of the observation value, and finally the optimized filtering parameters are used to realize more accurate motion trajectory estimation and filtering processing.

[0054] In the embodiments provided in the present application, by extracting and analyzing the motion feature and intensity of adjacent video frames, and dynamically adjusting the process noise and measurement noise covariance parameters of the filter, accurate modeling and noise suppression of the video motion state are realized, thereby effectively improving the robustness of the motion estimation and the adaptability of the filtering processing.

[0055] Based on the above embodiments, in one of the example embodiments provided in the present application, the specific implementation process of determining the motion feature corresponding to the video to be processed based on a plurality of video frames, and determining the motion intensity of the video to be processed based on the motion feature can further include steps S401 and S402, which are described in detail as follows:

[0056] Step S401: Determine the motion parameters corresponding to the plurality of video frames, and determine the corresponding motion feature based on the motion parameters. The motion feature includes a mean value and a covariance.

[0057] Step S402: Determine the motion intensity of the video to be processed based on the mean value and the covariance.

[0058] Exemplarily, stable and trackable feature points are extracted from the frames, and displacement vectors of the feature points between adjacent frames are calculated as motion parameters, which contain motion components of each feature point in x and y directions, reflecting local characteristics of inter-frame motion. Then, statistical analysis is performed on the displacement vectors of all feature points, and mean values of the displacement vectors are calculated, such as average values of motion vectors of all feature points and central tendency of overall motion for describing direction and amplitude of overall translation or rotation and covariance. The covariance matrix quantifies the variation degree of displacement vectors in x and y directions and the correlation between the two directions, and the larger the covariance value is, the higher the dispersion degree of motion in a certain direction or the existence of direction coupling motion is. The mean values can reflect the overall motion direction (such as left or up) and the average motion amplitude (such as displacement size) of the video, and the covariance can reveal the direction dispersion of motion. For example, whether there is multi-directional motion and the complexity of motion mode, specifically, whether there is composite motion such as rotation and scaling. Finally, the motion intensity is determined based on the comprehensive analysis of the mean values and the covariance. For example, by weighted combination of the modulus of the mean values and the trace of the covariance, or combining the eigenvalue analysis of the covariance matrix to determine the main direction and energy distribution of the motion, the motion intensity value is finally quantified.

[0059] In the embodiments provided in the present application, by calculating the mean value and the covariance based on the motion parameters of multiple frames to quantify the motion characteristics, the statistical characteristics of video motion can be accurately described, and the motion intensity can be effectively determined, thereby providing more reliable motion state basis for subsequent filtering and image stabilization processing.

[0060] Based on the above embodiments, in one of the exemplary embodiments provided in the present application, the specific implementation process of determining the scene type corresponding to the to-be-processed video can further include steps S501 and S502, which are described in detail as follows:

[0061] Step S501: determining an image feature statistical quantity variance corresponding to the to-be-processed video, the image feature statistical quantity variance including gradient entropy and brightness variance.

[0062] Step S502: determining the scene type corresponding to the to-be-processed video based on the gradient entropy and the brightness variance.

[0063] Exemplarily, in determining the scene type corresponding to the video to be processed, image feature statistics can be extracted from the video frames, wherein the gradient entropy is used to quantify the image texture complexity, and the brightness variance is used to measure the degree of light change. Specifically, the gradient entropy is obtained by calculating the probability distribution of the gradient amplitude of the pixel points in the image and applying the information entropy formula, and the higher the value, the richer the image texture and the more details (such as complex texture scenes), and the lower the value, the more single the texture (such as low-texture scenes such as walls and sky). The brightness variance is calculated by statistics of the variance of the brightness values of all pixel points in the image, and the larger the variance, the more dramatic the brightness fluctuation between frames or within frames, such as high light change scenes such as indoor and outdoor rapid switching, strong light and shadow area, and the smaller the variance, the more stable the brightness distribution. Subsequently, the gradient entropy and the brightness variance are used as input features, and a pre-set scene classification threshold or a trained classification model is combined. For example, a model based on support vector machine and random forest, which learns the mapping relationship between gradient entropy and brightness variance through a large amount of video data labeled with scene types, to determine the scene type. Specifically, when the gradient entropy is lower than the preset entropy threshold and the brightness variance is lower than the stable light threshold, it is determined as a low-texture and stable light scene. When the brightness variance is higher than the high light change threshold, regardless of the gradient entropy, it is determined as a high light change scene, and when the gradient entropy is high and the brightness variance is moderate, it is possible to determine as a normal texture scene with certain light fluctuation. Finally, through the double-index quantitative analysis based on the gradient entropy and the brightness variance, the video scene type (such as low-texture, high light change, normal texture and stable light, etc.) can be accurately identified, which provides a reliable basis for dynamically selecting feature extraction algorithms such as GFTT algorithm, ORB algorithm or SIFT algorithm, ensures that the feature extraction strategy is highly matched with the actual scene characteristics, and thus improves the robustness of feature matching and the video processing effect.

[0064] In the embodiments provided in the present application, by analyzing the statistical characteristics of image gradient entropy and brightness variance, the video scene type can be accurately identified, which provides adaptive basis for subsequent feature extraction, filter parameter adjustment and other processing steps, thereby improving the scene adaptability and effect stability of video processing.

[0065] Based on the above embodiments, in one of the exemplary embodiments provided in the present application, the specific implementation process of determining the motion feature corresponding to the video to be processed based on the target feature extraction strategy can further include steps S601 and S602, which are described in detail as follows:

[0066] Step S601, determining the down-sampling ratio of the video to be processed based on the scene type, and performing sampling processing on the video to be processed based on the down-sampling ratio to obtain the sampled video data.

[0067] In step S602, feature extraction is performed on the sampled video data based on a target feature extraction strategy to determine the motion feature corresponding to the video to be processed.

[0068] For example, the down-sampling ratio of the video can be dynamically adjusted based on the scene type determined in the above embodiment. For example, for a low-texture scene such as a sky or a wall, a high down-sampling ratio such as reducing to half of the original resolution or lower can be used to reduce the amount of calculation and avoid noise interference, because the feature points are sparse and less sensitive to details. For a high-illumination-change scene such as indoor-outdoor rapid switching and strong light and shadow area, a lower down-sampling ratio such as keeping the original resolution or slightly reducing can be used to ensure the accuracy of feature extraction, because more details need to be retained to capture the texture difference caused by illumination change. After the down-sampling ratio is determined, the video to be processed is down-sampled in the spatial domain or the time domain to generate the sampled video data, which reduces the data dimension while retaining the key scene information, and improves the efficiency of subsequent processing. Then, feature extraction is performed on the sampled video data based on a target feature extraction strategy. For example, in a low-texture scene, the ORB algorithm which is less dependent on texture is selected to extract corner points, and in a high-illumination-change scene, the SIFT algorithm which is invariant to illumination is selected to extract key points. The corresponding relationship between the feature points of adjacent frames is established through a feature matching algorithm to form a feature point pair set, and then the inter-frame motion feature is analyzed.

[0069] Optionally, the motion feature corresponding to the video to be processed can be determined by calculating the displacement vector of the feature point pair, counting the direction distribution of the motion, such as the trend and amplitude distribution of the main motion direction, and the length change of the motion vector.

[0070] In the embodiments provided in the present application, the down-sampling ratio is adaptively adjusted based on the scene type to optimize the calculation efficiency, and the target feature extraction strategy is combined to accurately extract the motion feature in the down-sampled data, which realizes the dual goals of scene adaptation processing and efficient capture of motion information, thereby improving the accuracy and real-time performance of video stabilization and dynamic analysis.

[0071] Based on the above embodiments, in one of the exemplary embodiments provided in the present application, the specific implementation process of the above video processing method can further include steps S701 and S702, which are described in detail as follows:

[0072] In step S701, the difference between the current frame rate of the video to be processed and the target frame rate, and the real-time load parameter of the video processing device are obtained.

[0073] Step S702, adjust the down-sampling ratio based on the difference value and the real-time load parameter, to sample the video to be processed based on the adjusted down-sampling ratio.

[0074] For example, the difference value between the current frame rate and the target frame rate can be accurately calculated. For example, when the current frame rate is 60 fps and the target frame rate is 30 fps, the difference value is 30 fps. The difference value directly reflects the deviation of the video processing speed from the expected target. At the same time, the device load parameters such as CPU / GPU usage, memory occupancy, etc. can be monitored in real time, which dynamically reflects the occupation of the computing resources of the device. Then, the difference value and the load parameter are combined to make a comprehensive decision. When the difference value is large, for example, the current frame rate is significantly higher than the target frame rate and the device load is low, the down-sampling ratio can be appropriately increased. Specifically, the resolution can be reduced or frames can be skipped, etc. to reduce the amount of data per frame or the number of frames to reduce the computational complexity of feature extraction and motion analysis in subsequent processing, thereby improving the processing efficiency. When the difference value is small but the device load is high (for example, the CPU usage is close to saturation), the down-sampling ratio can be increased to reduce the computational pressure of the device and avoid processing delay or lag due to insufficient resources. When the difference value is large and the load is high, the down-sampling ratio needs to be increased more significantly to release the device resources to the maximum extent while ensuring the basic processing needs.

[0075] Finally, the original video is spatially or temporally down-sampled according to the adjusted down-sampling ratio. For example, the resolution can be reduced by bilinear interpolation or frames can be extracted at a fixed interval to obtain the sampled video data. The sampled video data not only meets the target frame rate requirement but also matches the real-time load capacity of the device, ensuring that subsequent feature extraction, motion feature analysis, etc. can be executed in an efficient and stable computing environment, thereby improving the robustness and real-time performance of the overall video processing flow.

[0076] In the embodiments provided in the present application, the difference value between the current frame rate and the target frame rate and the real-time load parameter of the device are combined to dynamically adjust the down-sampling ratio, which realizes intelligent matching of processing efficiency and computing resources. The real-time requirements of video stabilization and dynamic analysis are guaranteed, and the performance degradation caused by device overload is avoided, thereby improving the adaptive ability and running stability of the video processing system.

[0077] Based on the above embodiments, in one of the example embodiments provided in the present application, the specific implementation process of determining the spatial transformation parameter of the video to be processed based on the filtered video data can further include steps S801 to S803, which are described in detail as follows:

[0078] Step S801, determine the motion amplitude of the video to be processed based on the filtered video data, and determine the cropping boundary parameter of the video to be processed based on the motion amplitude.

[0079] Exemplarily, key motion information is extracted from the filtered video frames, for example, by calculating displacement vector fields between adjacent frames through optical flow method or feature point matching algorithm, which can accurately reflect translation, rotation, scaling and other motion components of pixels or feature points between frames, and then the overall motion amplitude of the video is quantified by statistics of the maximum value of the modulus of all inter-frame displacement vectors, the direction distribution concentration and the motion pattern. Specifically, the maximum translation value can reflect the range of picture offset caused by device jitter, the rotation angle can reflect the degree of lens rotation or object rotation, and the scaling ratio can measure the amplitude of the change in picture distance. Subsequently, combined with the quantification results of the motion amplitude, the maximum boundary expansion or contraction caused by motion in the entire video sequence is analyzed, for example, when there is significant translation in a frame, the offset of the picture boundary after the translation relative to the original picture needs to be calculated, and rotation or scaling needs to consider the influence of the geometric deformation of the transformed picture on the boundary.

[0080] Finally, based on the motion range statistics results of all frames, such as the maximum displacement, the maximum rotation angle, the maximum scaling ratio, etc., the cropping boundary parameters are determined. In addition, by calculating the minimum circumscribed rectangle of each boundary of the transformed picture, that is, the minimum region containing all the effective pixels of the frames, the coordinates of the top-left corner and the bottom-right corner of the cropping are determined, or the width, height and offset of the cropped region are directly calculated, to ensure that the cropped video sequence can not only eliminate the inter-frame misalignment and black border problem caused by motion, but also completely retain the core content of the picture, while maintaining the visual coherence between frames, and finally output a stable and natural video sequence.

[0081] In step S802, motion smoothing parameters of the video to be processed are determined based on the filtered video data, and geometric transformation parameters of the video to be processed are determined based on the motion smoothing parameters.

[0082] The displacement vector fields between adjacent frames are extracted by the optical flow method or the feature point matching algorithm, which reflects the local characteristics of the inter-frame motion. Then, the displacement vectors can be processed in the time domain by using a smoothing algorithm such as Kalman filtering, low-pass filtering or moving average filtering, so as to suppress abnormal fluctuations caused by device jitter or noise and generate a smooth motion trajectory curve. The slope, curvature and fluctuation range of the motion trajectory curve can be used as motion smoothing parameters to quantify the continuity and stability of the motion. Then, the geometric transformation mode existing between the frames is analyzed based on the smoothed motion trajectory, for example, the overall direction distribution of the displacement vectors is counted by whether the translation is dominant or there is a rotation or scaling trend and amplitude variation rule, and in combination with the affine transformation model for describing translation, rotation and scaling, or the homography transformation model for processing perspective transformation, the geometric transformation parameters corresponding to the video are derived. The geometric transformation parameters include the translation amount of the overall displacement of the image plane, the rotation angle of the lens rotation or object rotation, the scaling ratio of the image plane distance change and the possible perspective transformation matrix elements, so as to ensure that the transformation parameters can accurately describe the inter-frame motion relationship.

[0083] Finally, the pixels misaligned between the frames are adjusted to the correct position by coordinate mapping and pixel resampling of each frame based on the geometric transformation parameters, so as to eliminate the inter-frame offset and black edge problem caused by motion, and at the same time, the core picture content is reserved in combination with the boundary clipping parameters, and finally a stable video sequence with visual coherence and stable motion is output.

[0084] In step S803, the spatial transformation parameters of the video to be processed are determined based on the clipping boundary parameters and the geometric transformation parameters.

[0085] For example, based on the clipping boundary parameters such as the coordinates and sizes of the clipping region and the geometric transformation parameters such as the translation amount, rotation angle, scaling ratio and perspective transformation matrix, a spatial transformation matrix integrating the coordinate mapping rule and the boundary limitation is constructed, and the matrix is processed in time sequence to suppress parameter fluctuations, and finally the matrix is applied to each frame for coordinate mapping and pixel resampling, so as to ensure that the inter-frame motion is accurately aligned and the picture content is complete and coherent, thereby determining the spatial transformation parameters of the video to be processed.

[0086] In the embodiments provided in the present application, the clipping boundary parameters are determined by quantifying the motion amplitude based on the filtered video data, and the geometric transformation parameters are optimized in combination with the motion smoothing parameters, so as to realize the collaborative optimization of clipping and geometric correction, thereby accurately constructing the spatial transformation parameters to improve the dynamic adaptability and visual stability of video stabilization.

[0087] Please refer to Figure 3 , Figure 3 The flowchart of the video processing method provided in the present application Figure 2 Please refer toFigure 3 In some possible embodiments, a variance of image feature statistics corresponding to the to-be-processed video is determined, the variance of image feature statistics including gradient entropy and brightness variance; a scene type corresponding to the to-be-processed video is determined based on the gradient entropy and the brightness variance. A downsampling ratio of the to-be-processed video is determined based on the scene type, and the to-be-processed video is sampled based on the downsampling ratio to obtain sampled video data; feature extraction is performed on the sampled video data based on a target feature extraction strategy to determine motion features corresponding to the to-be-processed video. The target feature extraction strategy of the to-be-processed video is determined based on the scene type; a plurality of video frames of the to-be-processed video are extracted based on the target feature extraction strategy, the plurality of video frames being adjacent video frames in the to-be-processed video; motion features corresponding to the to-be-processed video are determined based on the plurality of video frames, and a motion intensity of the to-be-processed video is determined based on the motion features; filter parameters of a target filter are determined based on the motion intensity, the filter parameters including process noise covariance and measurement noise covariance. The filter parameters of the to-be-processed video are determined based on the motion intensity, and the to-be-processed video is filtered based on the filter parameters to obtain filtered video data; spatial transformation parameters corresponding to the to-be-processed video are determined based on the filtered video data, and interframe transformation is performed on the to-be-processed video based on the spatial transformation parameters to obtain a stable video sequence. For detailed implementation process, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0088] Figure 4 A structural schematic diagram of a video processing apparatus provided in the present application is shown in FIG. 4. As shown in FIG. 4, the video processing apparatus 40 provided in the present embodiment includes a determination module 410, a feature extraction module 420, a filtering module 430, and a transformation module 440. Figure 4 The determination module 410 is configured to determine a scene type corresponding to a to-be-processed video, and determine a target feature extraction strategy of the to-be-processed video based on the scene type. The feature extraction module 420 is configured to determine motion features corresponding to the to-be-processed video based on the target feature extraction strategy, and determine a motion intensity of the to-be-processed video based on the motion features. The filtering module 430 is configured to determine filter parameters of a target filter based on the motion intensity, and perform filtering on the to-be-processed video based on the filter parameters to obtain filtered video data. The transformation module 440 is configured to determine spatial transformation parameters corresponding to the to-be-processed video based on the filtered video data, and perform interframe transformation on the to-be-processed video based on the spatial transformation parameters to obtain a stable video sequence.

[0089] In a possible implementation, the filtering module 430 is further configured to extract a plurality of video frames of the to-be-processed video based on the target feature extraction strategy, the plurality of video frames being adjacent video frames in the to-be-processed video; determine motion features corresponding to the to-be-processed video based on the plurality of video frames, and determine a motion intensity of the to-be-processed video based on the motion features; determine filter parameters of a target filter based on the motion intensity, the filter parameters including process noise covariance and measurement noise covariance.

[0090] In one possible implementation, the filtering module 430 is further configured to determine motion parameters corresponding to multiple video frames, and determine corresponding motion features based on the motion parameters, the motion features including mean and covariance; and determine the motion intensity of the video to be processed based on the mean and covariance.

[0091] In one possible implementation, the determining module 410 is further configured to determine the variance of image feature statistics corresponding to the video to be processed, the variance of image feature statistics including gradient entropy and brightness variance; and determine the scene type corresponding to the video to be processed based on gradient entropy and brightness variance.

[0092] In one possible implementation, the feature extraction module 420 is further configured to: determine the downsampling ratio of the video to be processed based on the scene type, and perform sampling processing on the video to be processed based on the downsampling ratio to obtain sampled video data; and perform feature extraction on the sampled video data based on the target feature extraction strategy to determine the motion features corresponding to the video to be processed.

[0093] In one possible implementation, the feature extraction module 420 is further configured to: obtain the difference between the current frame rate and the target frame rate of the video to be processed, and the real-time load parameters of the video processing device; adjust the downsampling ratio based on the difference and the real-time load parameters, and perform sampling processing on the video to be processed based on the adjusted downsampling ratio.

[0094] In one possible implementation, the transformation module 440 is further configured to: determine the motion amplitude of the video to be processed based on the filtered video data, and determine the cropping boundary parameters of the video to be processed based on the motion amplitude; determine the motion smoothing parameters of the video to be processed based on the filtered video data, and determine the geometric transformation parameters of the video to be processed based on the motion smoothing parameters; and determine the spatial transformation parameters of the video to be processed based on the cropping boundary parameters and the geometric transformation parameters.

[0095] The video processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0096] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 510 and a memory 520. Optionally, the device 50 further includes a communication component 530. The processor 510, memory 520, and communication component 530 are connected via a bus 540.

[0097] In the implementation process, the at least one processor 510 executes the computer-executable instructions stored in the memory 520, so that the at least one processor 510 performs the above-mentioned method.

[0098] The specific implementation process of the processor 510 can refer to the method embodiments described above, which have similar implementation principles and technical effects, and will not be described here in detail.

[0099] In the above embodiments, it should be understood that the processor can be a central processing unit (English: Central Processing Unit, CPU for short), and can also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, DSP for short), application specific integrated circuits (English: Application Specific Integrated Circuit, ASIC for short), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as hardware processor execution or combined with hardware and software modules in the processor for execution.

[0100] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0101] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0102] The present application also provides a computer program product, comprising a computer program, which is executed by a processor to implement the above-mentioned method.

[0103] The present application also provides a computer-readable storage medium, which stores computer-executable instructions, and when the processor executes the computer-executable instructions, the above-mentioned method is implemented.

[0104] The above-mentioned readable storage medium can be realized by any type of volatile or nonvolatile storage devices or their combinations, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0105] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0106] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0108] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0109] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0110] It can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various media that can store program codes.

[0111] Finally, it should be noted that: those skilled in the art will easily think of other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application that follow the general principles of the present application and include common knowledge or conventional technical means in the art that are not disclosed in the present application, and is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims.

Claims

1. A method of video processing, the method comprising: The method comprises: determining a scene type corresponding to a to-be-processed video, determining a target feature extraction strategy of the to-be-processed video based on the scene type; determining a motion feature corresponding to the to-be-processed video based on the target feature extraction strategy, and determining a motion intensity of the to-be-processed video based on the motion feature; determining a filtering parameter of the to-be-processed video based on the motion intensity, and performing filtering processing on the to-be-processed video based on the filtering parameter to obtain filtered video data; determining a spatial transformation parameter corresponding to the to-be-processed video based on the filtered video data, and performing inter-frame transformation on the to-be-processed video based on the spatial transformation parameter to obtain a stable video sequence.

2. The method of claim 1, wherein, The method further comprises: extracting a plurality of video frames of the to-be-processed video based on the target feature extraction strategy, the plurality of video frames being adjacent video frames in the to-be-processed video; determining a motion feature corresponding to the to-be-processed video based on the plurality of video frames, and determining a motion intensity of the to-be-processed video based on the motion feature; determining a filtering parameter of a target filter based on the motion intensity, the filtering parameter including process noise covariance and measurement noise covariance.

3. The method of claim 2, wherein, The determination of the motion feature corresponding to the to-be-processed video based on the plurality of video frames and the determination of the motion intensity of the to-be-processed video based on the motion feature comprise: determining a motion parameter corresponding to the plurality of video frames, and determining a corresponding motion feature based on the motion parameter, the motion feature including a mean value and a covariance; determining the motion intensity of the to-be-processed video based on the mean value and the covariance.

4. The method according to any one of claims 1 to 3, characterized in that, The determination of the scene type corresponding to the to-be-processed video comprises: determining an image feature statistical variance corresponding to the to-be-processed video, the image feature statistical variance including a gradient entropy and a brightness variance; determining the scene type corresponding to the to-be-processed video based on the gradient entropy and the brightness variance.

5. The method of claim 4, wherein, The determination of the motion feature corresponding to the to-be-processed video based on the target feature extraction strategy comprises: determining a downsampling ratio of the to-be-processed video based on the scene type, and performing sampling processing on the to-be-processed video based on the downsampling ratio to obtain sampled video data; performing feature extraction on the sampled video data based on the target feature extraction strategy to determine the motion feature corresponding to the to-be-processed video.

6. The method of claim 5, wherein, The method further comprises: obtaining a difference value between a current frame rate and a target frame rate of the to-be-processed video, and a real-time load parameter of a video processing device; adjusting the downsampling ratio based on the difference value and the real-time load parameter to perform sampling processing on the to-be-processed video based on the adjusted downsampling ratio.

7. The method according to any one of claims 1 to 3, wherein The determination of the spatial transformation parameter corresponding to the to-be-processed video based on the filtered video data comprises: determining a motion amplitude of the to-be-processed video based on the filtered video data, and determining a cropping boundary parameter of the to-be-processed video based on the motion amplitude; determining a motion smoothing parameter of the to-be-processed video based on the filtered video data, and determining a geometric transformation parameter of the to-be-processed video based on the motion smoothing parameter; Determine a spatial transformation parameter of the to-be-processed video based on the cropping boundary parameter and the geometric transformation parameter.

8. A video processing apparatus, comprising: The device comprises: A determination module is configured to determine a scene type corresponding to a to-be-processed video, and determine a target feature extraction strategy of the to-be-processed video based on the scene type. A feature extraction module is configured to determine a motion feature corresponding to the to-be-processed video based on the target feature extraction strategy, and determine a motion intensity of the to-be-processed video based on the motion feature. A filtering module is configured to determine a filtering parameter of the to-be-processed video based on the motion intensity, and perform filtering processing on the to-be-processed video based on the filtering parameter to obtain filtered video data. A transformation module is configured to determine a spatial transformation parameter corresponding to the to-be-processed video based on the filtered video data, and perform inter-frame transformation on the to-be-processed video based on the spatial transformation parameter to obtain a stable video sequence.

9. An electronic device, comprising: Comprise: A memory, a processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory, so that the processor executes the method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1 to 7.

Images