Video smoothness evaluation method, device and computer equipment

Abstract

The embodiment of the application discloses a video fluency evaluation method and device and computer equipment. Specifically, first, a target video is acquired, the target video includes multiple video frames, feature extraction is performed on the video frames to obtain multiple features of the video frames and motion state information of the features, a matching relationship of the features in adjacent video frames is calculated to obtain a motion state information set of each feature, the motion state information set includes motion state information of the features in different video frames, fluency evaluation is performed on the features according to the motion state information set to obtain motion fluency of each feature, and finally, multiple motion fluencies are aggregated to obtain video fluency of the target video. The embodiment of the application can significantly improve the efficiency and accuracy of video fluency evaluation.

Description

Technical Field

[0001] This application relates to the field of audio and video, specifically to a method, apparatus, and computer device for evaluating video smoothness. Background Technology

[0002] With technological advancements, video, as a more widespread medium of communication, is playing an increasingly important role in people's daily lives. Video generation methods are also becoming more diverse. For example, users can shoot videos using cameras, and then the videos can be automatically generated through generative algorithms. However, the quality of these massive amounts of video varies greatly. Video smoothness, as a crucial factor in measuring video quality, is currently mostly assessed manually, which is extremely inefficient and lacks accuracy. Summary of the Invention

[0003] This application provides a video smoothness evaluation method, apparatus, and computer device, which can significantly improve the efficiency and security of video smoothness evaluation.

[0004] This application provides a method for evaluating video smoothness, including:

[0005] Acquire the target video, which includes multiple video frames;

[0006] Feature extraction is performed on video frames to obtain multiple features of the video frames, as well as the motion state information of the features;

[0007] Calculate the matching relationship of features in adjacent video frames to obtain the motion state information set of each feature. The motion state information set includes the motion state information of the feature in different video frames.

[0008] The smoothness of each feature is evaluated based on the motion state information set to obtain the motion smoothness of each feature.

[0009] Multiple motion smoothness scores are aggregated to obtain the video smoothness score of the target video.

[0010] Accordingly, this application also provides a video smoothness evaluation device, including:

[0011] The acquisition module is used to acquire the target video, which includes multiple video frames.

[0012] The feature module is used to extract features from video frames, obtaining multiple features of the video frame and the motion state information of the features;

[0013] The calculation module is used to calculate the matching relationship of features in adjacent video frames and obtain the motion state information set of each feature. The motion state information set includes the motion state information of the feature in different video frames.

[0014] The evaluation module is used to evaluate the smoothness of features based on the motion state information set, and obtain the motion smoothness of each feature.

[0015] The aggregation module is used to aggregate multiple motion smoothness scores to obtain the video smoothness score of the target video.

[0016] In some embodiments of this application, multiple video frames correspond to different moments in the target video, and the motion state information set includes the motion state information of features at different moments.

[0017] The evaluation module may include:

[0018] The determination submodule is used to determine the state prediction information at different times through the target evaluation model;

[0019] The evaluation submodule is used to evaluate the motion smoothness of features based on state prediction information and the motion state information at the corresponding time.

[0020] In some embodiments of this application, the moments in the motion state information set correspond to at least two different dimensions of motion state information, and each dimension corresponds to a target evaluation model.

[0021] The evaluation submodule may include:

[0022] The evaluation unit is used to evaluate the motion smoothness of features in different dimensions based on state prediction information and the motion state information at the corresponding time.

[0023] The aggregation unit is used to aggregate motion smoothness in at least two different dimensions to obtain the motion smoothness of the features.

[0024] In some embodiments of this application, the dimension includes a positional dimension with a defined direction, and the evaluation unit can be specifically used for:

[0025] By using the motion state information at different times under the position dimension of a set direction, the movement distance of the feature in the set direction within the first time interval is calculated;

[0026] Based on the distance traveled and the state prediction and motion state information within the first time interval, the motion smoothness of the feature in the position dimension of the set direction is evaluated.

[0027] In some embodiments of this application, the video smoothness evaluation apparatus further includes an initial evaluation module and a target evaluation module, wherein,

[0028] The initial evaluation module is used to determine the initial evaluation model for the features;

[0029] The target evaluation module is used to optimize the initial evaluation model based on the motion state information set to obtain a target evaluation model with features.

[0030] In some embodiments of this application, the initial evaluation model includes multiple initial parameters, and the target evaluation module can be specifically used for:

[0031] Input multiple time points and multiple motion state information into the initial evaluation model to obtain the first output value;

[0032] If the first output value does not match the preset output value, or the number of parameter updates of the initial evaluation model is less than the preset number, update the initial parameters to obtain an updated evaluation model, determine the updated evaluation model as the new initial evaluation model, and return to execute the step of inputting multiple time moments and multiple motion state information into the initial evaluation model to obtain the first output value;

[0033] If the first output value matches the preset output value, or the number of parameter updates of the initial evaluation model is equal to the preset number, the initial evaluation model is determined as the target evaluation model with the following features.

[0034] In some embodiments of this application, the target video includes a video uploaded by a user on a streaming media platform, and the video smoothness evaluation device further includes a frame rate adjustment module, which is specifically used for:

[0035] If the video smoothness is greater than the first preset smoothness, reduce the frame rate of the target video to obtain the first processed video;

[0036] If the video smoothness is lower than the second preset smoothness and the target video carries a hotspot video identifier, the frame rate of the target video is increased to obtain the second processed video.

[0037] In some embodiments of this application, the target video includes a video generated by a generative video platform based on a first video generation algorithm. The generative video platform also includes at least one second video generation algorithm. The video smoothness evaluation device further includes a generation adjustment module, which is specifically used for:

[0038] If the video smoothness is less than the third preset smoothness corresponding to the first video generation algorithm, the generative video platform is triggered to regenerate the target video based on the second video generation algorithm.

[0039] Accordingly, this application also provides a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the video smoothness evaluation method provided in this application.

[0040] Accordingly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps in the video smoothness evaluation method provided in embodiments of this application.

[0041] Accordingly, embodiments of this application also provide a computer program product, including a computer program or instructions, which are executed by a processor to implement the steps in the video smoothness evaluation method provided in embodiments of this application.

[0042] This application embodiment can acquire a target video, which includes multiple video frames. Feature extraction is performed on the video frames to obtain multiple features and motion state information of the features. The matching relationship between features in adjacent video frames is calculated to obtain a motion state information set for each feature. This motion state information set includes the motion state information of the feature in different video frames. Smoothness evaluation is performed on the features based on the motion state information set to obtain the motion smoothness of each feature. Multiple motion smoothness values ​​are aggregated to obtain the video smoothness of the target video. This application can perform smoothness calculation at the feature level in a more refined manner, and then aggregate the motion smoothness values ​​of all features to obtain a more accurate video smoothness of the target video. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of a scenario for the video smoothness evaluation method provided in the embodiments of this application;

[0045] Figure 2 This is a flowchart illustrating the video smoothness evaluation method provided in the embodiments of this application;

[0046] Figure 3 This is a video frame diagram illustrating the video smoothness evaluation method provided in the embodiments of this application;

[0047] Figure 4 This is a schematic diagram illustrating the features of the video smoothness evaluation method provided in the embodiments of this application;

[0048] Figure 5 This is a schematic diagram illustrating the matching of features in the video smoothness evaluation method provided in this application embodiment;

[0049] Figure 6 This is a schematic diagram illustrating the characteristic state information of the video smoothness evaluation method provided in the embodiments of this application;

[0050] Figure 7 This is a flowchart of the model optimization of the video smoothness evaluation method provided in the embodiments of this application;

[0051] Figure 8 This is another flowchart illustrating the video smoothness evaluation method provided in the embodiments of this application;

[0052] Figure 9 This is a schematic diagram of the video smoothness evaluation device provided in the embodiments of this application;

[0053] Figure 10 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0056] This application provides a video smoothness evaluation method, apparatus, and computer device. The video smoothness evaluation apparatus can be integrated into a video smoothness evaluation system, and the video smoothness evaluation system can be integrated into at least one computer device, which may include at least one of a terminal and a server.

[0057] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not restricted herein.

[0058] The video smoothness evaluation system can be integrated into a server. The server can acquire a target video, which includes multiple video frames. Feature extraction is performed on the video frames to obtain multiple features of the video frames and the motion state information of the features. The matching relationship of features in adjacent video frames is calculated to obtain a motion state information set for each feature. The motion state information set includes the motion state information of the feature in different video frames. Smoothness evaluation is performed on the features based on the motion state information set to obtain the motion smoothness of each feature. Multiple motion smoothness values ​​are aggregated to obtain the video smoothness of the target video.

[0059] For example, see Figure 1 The video smoothness evaluation system may include a terminal 10 and a server 20. The terminal 10 can transmit a target video 1 uploaded by the user to the server 20. The server 20 can extract features from multiple video frames in the target video 1 to obtain multiple features of the video frames and the motion state information of the features. Then, it can calculate the matching relationship of features in adjacent video frames to obtain a motion state information set for each feature. The motion state information set includes the motion state information of the feature in different video frames. Finally, it evaluates the smoothness of the features based on the motion state information set to obtain the motion smoothness of each feature. The multiple motion smoothness scores are aggregated to obtain the video smoothness score 'a' of the target video 1. The server 20 can process the target video based on the video smoothness score 'a' and transmit the processed video to other computer devices.

[0060] Figure 1 This is an example of an application scenario for the video smoothness evaluation system of this application. It is mainly used to introduce, and not to limit, the video smoothness evaluation system of this application. In the actual application of the technical solutions described in the embodiments of this application, the computer equipment included in the video smoothness evaluation system and the steps performed by each computer equipment can be flexibly adjusted, and it is not limited to... Figure 1 The content described in the text.

[0061] The video smoothness evaluation method of this application will be further described below with reference to embodiments. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments. The video smoothness evaluation method can be integrated into a video smoothness evaluation device, and the video smoothness evaluation device can be integrated into a video smoothness evaluation system. Figure 2 A flowchart illustrating the video smoothness evaluation method of this application is shown, as follows: Figure 2 Video smoothness evaluation methods may include:

[0062] 110. Obtain the target video, which includes multiple video frames.

[0063] The target video can include videos whose smoothness needs to be evaluated. The source of the target video can be various, such as: videos pre-stored locally on the computer device, videos generated by the computer device based on a preset algorithm (such as video generation algorithms), videos received by the computer device from other devices (such as videos uploaded by users through mobile terminals), and so on.

[0064] In this embodiment, the acquisition process can include various methods. For example, a video acquisition request can be output first, including the identifier information of the target video, and then the target video returned based on the video acquisition request can be received, such as a video stored locally on the computer device or on another computer device, or a generated video. Alternatively, target videos transmitted from other computer devices, such as videos uploaded by users, can be received at any time.

[0065] In this context, a video frame is a unit of the target video. A video frame is in image format, and multiple video frames constitute the target video. Each video frame corresponds to a specific moment in the target video; for example, see [link to relevant documentation]. Figure 3 Video 1 consists of multiple consecutive video frames, including video frames 1-3.

[0066] 120. Perform feature extraction on video frames to obtain multiple features of the video frames and the motion state information of the features.

[0067] The features can include the content presented in a certain area of ​​a video frame. The features can include specific objects in a certain area, such as leaves, eyes, faces, water droplets, mountains, the moon, etc. The features can also include specific colors (or combinations of colors), specific light and dark relationships (or combinations of light and dark relationships), specific lines (or combinations of lines), etc., presented in a certain area. The content presented in the certain area where the features are located will vary in different application scenarios, and can be flexibly analyzed for specific scenarios.

[0068] Among them, motion state information includes information reflecting the motion state of the feature in its video frame, which may include the feature's position, size, orientation, etc. in the video frame.

[0069] Specifically, there are several ways to extract features from video frames to obtain their features and motion state information. For example, one can first convert the video frame to grayscale to obtain the grayscale value of each pixel, then normalize all pixel grayscale values, calculate the gradient of each pixel, divide the image into multiple sub-blocks, each sub-block containing multiple pixels, and generate gradient information for each image block based on the pixel gradients. Several sub-blocks are then combined to form an image block, and the gradient information of all sub-blocks within an image block is concatenated to form the feature information of that image block. This feature information is then matched against the feature information of several preset features, and the preset feature with the highest matching degree is determined as the feature of that image block. Alternatively, feature extraction algorithms such as Scale-invariant Feature Transform (SIFT) and Speed-Up Robust Feature (SURF) can be used. See [link to relevant documentation]. Figure 4 Feature extraction was performed on video frame 1 to obtain multiple features and motion state information for each feature. Figure 4 Each circle represents a feature and its motion state information. The center point of the circle represents the center position of the feature, the area of ​​the circle represents the size of the feature, and the line connecting the center point of the circle to the circumference (radius) represents the direction of motion of the feature, such as feature 1 and feature 2 shown in the figure.

[0070] 130. Calculate the matching relationship of features in adjacent video frames to obtain the motion state information set of each feature. The motion state information set includes the motion state information of the feature in different video frames.

[0071] To aggregate the motion state information of features across different video frames, feature matching calculations can be performed on features in adjacent video frames to determine the corresponding features of a feature in one video frame in another video frame, thereby obtaining the corresponding features of the feature in other video frames and forming a set of feature motion state information.

[0072] Specifically, matching calculations can take many forms. For example, if video frame 1 and video frame 2 are adjacent video frames, feature 1A in video frame 1 can be compared with all features in video frame 2 to obtain the similarity of each feature, and the feature 2C with the highest similarity is determined as the matching feature of feature 1A. Matching algorithms can also be used for matching calculations, such as the Fast Library for Approximate Nearest Neighbors (FLANN). For example, see [link to relevant documentation]. Figure 5By performing matching calculations on video frame 2 and video frame 3, we can obtain the corresponding features of the features in video frame 2 in video frame 3. For example, feature a2 in video frame 2 corresponds to feature a3 in video frame 3.

[0073] By performing matching calculations on all adjacent video frames, a motion state information set for each feature can be obtained. This set includes the motion state information of the feature in different video frames, with each frame corresponding to a different moment in the target video. Therefore, the motion state information set of the video frames can be used to obtain, for example, [the following]. Figure 6 The diagram shows the position of feature p at different times.

[0074] 140. Evaluate the smoothness of features based on the motion state information set to obtain the motion smoothness of each feature.

[0075] Among them, motion smoothness can quantify the smoothness of feature changes in a video. There are many ways to evaluate smoothness. For example, based on the motion state information set, the change information of motion state information within a unit time period can be determined, and the motion smoothness can be determined based on the change information, the attribute of the feature (such as its category), and the attribute of the target video. For example, if the difference between the trend of the change information of the animal's ear in the animal video and the preset trend is less than the preset first threshold, the motion smoothness of the animal's ear can be determined as the smoothness 1 corresponding to the preset first threshold.

[0076] In some embodiments of this application, the state prediction information of a feature at different times can be determined by a target evaluation model, and then the motion smoothness of the feature can be jointly evaluated based on the state prediction information and motion state information of the feature.

[0077] The target evaluation model here can be used to determine the motion prediction information of features, which includes information obtained by predicting the motion state of features at a certain moment.

[0078] Features can include multiple motion state information belonging to different dimensions. The target evaluation model can predict the motion state information of the feature in multiple dimensions. For example, through the target evaluation model, the predicted motion direction information, predicted position information, and predicted size information of feature 1 can be obtained simultaneously. Embodiments of this application can also determine target evaluation models for different dimensions and determine the motion prediction information of that feature dimension through the target evaluation model for that specific dimension. For example, a direction evaluation model, a position evaluation model, and a size evaluation model can be determined separately. Then, the predicted motion direction of feature 2 can be obtained through the direction evaluation model, the predicted motion direction of feature 2 can be obtained through the position evaluation model, and the predicted size of feature 2 can be obtained through the size evaluation model.

[0079] In this embodiment, an initial evaluation model for a feature / a certain dimension of the feature can be determined first, and then the initial evaluation model can be optimized based on the motion state information set of the feature to obtain the target evaluation model for the feature / a certain dimension of the feature. Specifically, the evaluation model can be a deep learning / machine learning model, or a mathematical equation (such as a quadratic equation), etc., which can be flexibly selected according to actual needs.

[0080] The initial evaluation model can include multiple initial parameters. The optimization process of the initial evaluation model can be based on optimization algorithms (such as gradient descent or Newton's method) to optimize the initial parameters. The optimization termination condition can be, for example, that the error value is lower than the error threshold, or that the number of parameter optimizations exceeds the number threshold, etc. The selection of the optimization termination condition, the setting of the error threshold, and the number threshold can all be flexibly handled by the developers based on the actual scenario requirements and processing experience, and are not restricted here.

[0081] The process of optimizing the initial evaluation model may include: inputting multiple time points and multiple motion state information into the initial evaluation model to obtain a first output value; if the first output value does not match a preset output value, or the number of parameter updates of the initial evaluation model is less than a preset number, updating the initial parameters to obtain an updated evaluation model, determining the updated evaluation model as the new initial evaluation model, and returning to the step of inputting multiple time points and multiple motion state information into the initial evaluation model to obtain a first output value; if the first output value matches a preset output value, or the number of parameter updates of the initial evaluation model is equal to a preset number, determining the initial evaluation model as the target evaluation model for the features.

[0082] For example, a quadratic equation can be used to model a certain dimension (such as position, size, direction, etc.) to obtain a motion model for that dimension. If the motion process that conforms to the characteristics of the motion model is smooth, then the closer the motion state information in the motion state information set is to the motion model, the higher the smoothness of the motion of that feature.

[0083] For example, a quadratic equation f(t) = at can be constructed for the horizontal motion dimension. 2 +bt+c, where a, b, c are initial parameters, t is time t, and f(t) is the horizontal position of the feature at time t. The objective function is S(a, b, c) = ∑(f(t) +bt+c). i )-x i ) 2 The motion state information set includes different times t i Horizontal position x i .

[0084] The parameters p = (a, b, c) can be iteratively optimized using Newton's method. The optimization process stops when the error S is less than a preset error threshold, or when the number of iterations meets a preset iteration threshold, yielding the target evaluation model f(t) with the target parameter p. This process can be described as follows: Figure 7 As shown.

[0085] In this embodiment of the application, if state prediction information of different dimensions at different times is obtained through multiple target evaluation models, and then the motion smoothness of the feature in different dimensions is obtained, then all motion smoothness in different dimensions can be aggregated to obtain the motion smoothness of the feature.

[0086] For example, by using target evaluation models 1-3, the state prediction information of the feature at different times in the horizontal position dimension, direction dimension, and size dimension is obtained. Then, by combining the motion state information of the feature at different times in the horizontal position dimension, direction dimension, and size dimension, the motion smoothness of the feature in the horizontal position dimension, direction dimension, and size dimension is obtained respectively. Finally, the motion smoothness of 1-3 is aggregated to obtain the motion smoothness of the feature O.

[0087] Specifically, the motion smoothness of a feature can be obtained by aggregating all motion smoothness across different dimensions using averaging functions, median functions, etc. For example, the average of all motion smoothness across different dimensions can be calculated, and the resulting average value can be used as the video smoothness of that feature. The method of averaging can be flexibly adjusted according to the actual application scenario. For example, all motion smoothness values ​​can be summed, and then the sum can be divided by the number of dimensions to obtain the average value. Alternatively, weights can be assigned to different dimensions, and a weighted sum can be performed to obtain the average value.

[0088] In this embodiment of the application, if state prediction information of different dimensions at different times is obtained through multiple target evaluation models, and the motion smoothness of the feature in different dimensions is obtained, the motion smoothness of the feature in different dimensions can also be evaluated and filtered to determine the target motion smoothness of a certain dimension that best reflects the overall smoothness of the feature in all dimensions, and this is taken as the motion smoothness of the feature.

[0089] There are several ways to select a specific smoothness from multiple smoothness levels across different dimensions. For example, multiple motion smoothness levels can be sorted, and the motion smoothness level corresponding to a specific sorting result can be selected. Specific sorting results can include middle order, maximum order, minimum order, etc.

[0090] For example, by using the motion state information of the target evaluation model 1-3 and the motion state information of the feature at different times in the horizontal position dimension, direction dimension and size dimension, the motion smoothness 1-3 of the feature in the horizontal position dimension, direction dimension and size dimension are obtained respectively. The motion smoothness 1-3 is sorted and the motion smoothness 1 in the horizontal position dimension in the middle of the shot is determined as the motion smoothness of the feature.

[0091] Specifically, there are several ways to evaluate the motion smoothness of a feature in a certain dimension using state prediction information and motion state information. For example, one could calculate the difference between the state prediction information and motion state information at the same time point, and then sum all the differences using weighted averages to obtain the motion smoothness of the feature in that dimension. Another approach is to calculate the difference between the state prediction information and motion state information at the same time point, determine the smoothness points corresponding to each difference from a mapping table containing the differences and smoothness points, and then sum all the smoothness points to obtain the motion smoothness of the feature in that dimension. This mapping table can be pre-set by the developers.

[0092] This application embodiment can also take the position dimension of a set direction as an example. The movement distance of the feature in the set direction within the first time interval can be calculated by using the motion state information at different times under the position dimension of the set direction. Then, based on the movement distance, the state prediction information and motion state information within the first time interval, the smoothness of the feature's movement under the position dimension of the set direction can be evaluated.

[0093] The orientation here can be horizontal or vertical, and the first time interval can include a time interval consisting of two moments separated by N (N is an integer greater than or equal to 0) data frames.

[0094] Positions at different times: (t1, x1), (t2, x2), ..., (t N x N )

[0095] Model error:

[0096] t i Displacement at time: Δx i =|x i+1 -x i |

[0097] Distance:

[0098] The motion smoothness of the feature in the horizontal direction can then be S / L. To facilitate subsequent calculations, normalization can be performed to obtain the motion smoothness of the feature in the horizontal direction after normalization.

[0099] 150. Aggregate multiple motion smoothness scores to obtain the video smoothness score of the target video.

[0100] In this embodiment, all motion smoothness can be directly aggregated to obtain a more comprehensive video smoothness containing all feature information. Alternatively, multiple motion smoothness can be analyzed and filtered first, and some motion smoothness that does not represent the smoothness of the video and is not significant can be removed. Then, the remaining motion smoothness can be aggregated to obtain a more refined video smoothness.

[0101] Specifically, multiple motion smoothness metrics can be aggregated using averaging functions, median functions, etc., to obtain the video smoothness of the target video. For example, the average of all motion smoothness metrics can be calculated, and the average value can be used as the video smoothness. The averaging method can be flexibly handled according to the actual application scenario. For example, the sum of all motion smoothness metrics can be obtained by dividing the sum by the number of motion smoothness metrics. Alternatively, weights can be assigned to different motion smoothness metrics, and a weighted sum can be performed to obtain the average value. Or, all motion smoothness metrics can be sorted, and the median can be determined and used as the video smoothness.

[0102] As video generation algorithms become increasingly popular, how to efficiently and conveniently evaluate their quality has become a problem. The video smoothness evaluation method in this application can efficiently and quickly evaluate videos generated by video generation algorithms, effectively improving algorithm development efficiency and reducing the cost of manual evaluation.

[0103] In addition, the video smoothness evaluation method of this application can also be used in streaming media platforms. It can evaluate target videos uploaded by users. If the video smoothness of the target video is higher than the first preset smoothness, the frame rate of the target video is reduced to reduce the operating cost of the streaming media platform. If the video smoothness of the target video is lower than the preset second smoothness and carries a hot video identifier, the frame rate of the target video can be increased to improve the quality of the potential hot video and facilitate its better dissemination on the streaming media platform.

[0104] The video smoothness evaluation method of this application will be further described below with reference to embodiments. Specifically, as shown in the following examples... Figure 8 As shown:

[0105] 210. Obtain the target video.

[0106] For example, the target video is a video uploaded by a user on a short video platform, which is a cute pet short video shot by the user with their mobile phone.

[0107] 220. Calculate the features of the video frames to obtain the motion state of each feature.

[0108] 230. Calculate the feature matching relationship between different frames to obtain the motion state of the features at different times.

[0109] 240. Select a model to fit the motion state of each feature.

[0110] For example, a motion model can output the predicted motion state of features at different times.

[0111] 250. Determine the motion smoothness of each feature based on the difference between the model prediction and the actual motion state.

[0112] 260. Determine the smoothness of the target video based on the motion smoothness of each feature.

[0113] This application embodiment can acquire a target video, which includes multiple video frames. Feature extraction is performed on the video frames to obtain multiple features and motion state information of the features. The matching relationship between features in adjacent video frames is calculated to obtain a motion state information set for each feature. This motion state information set includes the motion state information of the feature in different video frames. Smoothness evaluation is performed on the features based on the motion state information set to obtain the motion smoothness of each feature. Multiple motion smoothness values ​​are aggregated to obtain the video smoothness of the target video. This application can perform smoothness calculation at the feature level in a more refined manner, and then aggregate the motion smoothness values ​​of all features to obtain a more accurate video smoothness of the target video.

[0114] To better implement the above methods, embodiments of this application also provide a video smoothness evaluation device, such as... Figure 9 As shown, the video smoothness evaluation device may include an acquisition module 310, a feature module 320, a calculation module 330, an evaluation module 340, and an aggregation module 350, wherein,

[0115] The acquisition module 310 is used to acquire a target video, which includes multiple video frames.

[0116] The feature module 320 is used to extract features from video frames to obtain multiple features of the video frames and motion state information of the features.

[0117] The calculation module 330 is used to calculate the matching relationship of features in adjacent video frames and obtain the motion state information set of each feature. The motion state information set includes the motion state information of the feature in different video frames.

[0118] The evaluation module 340 is used to evaluate the smoothness of features based on the motion state information set, and obtain the motion smoothness of each feature.

[0119] The aggregation module 350 is used to aggregate multiple motion smoothness metrics to obtain the video smoothness of the target video.

[0120] In some embodiments of this application, multiple video frames correspond to different moments in the target video, and the motion state information set includes the motion state information of features at different moments.

[0121] The evaluation module may include:

[0122] The determination submodule is used to determine the state prediction information at different times through the target evaluation model;

[0123] The evaluation submodule is used to evaluate the motion smoothness of features based on state prediction information and the motion state information at the corresponding time.

[0124] In some embodiments of this application, the moments in the motion state information set correspond to at least two different dimensions of motion state information, and each dimension corresponds to a target evaluation model.

[0125] The evaluation submodule may include:

[0126] The evaluation unit is used to evaluate the motion smoothness of features in different dimensions based on state prediction information and the motion state information at the corresponding time.

[0127] The aggregation unit is used to aggregate motion smoothness in at least two different dimensions to obtain the motion smoothness of the features.

[0128] In some embodiments of this application, the dimension includes a positional dimension with a defined direction, and the evaluation unit can be specifically used for:

[0129] By using the motion state information at different times under the position dimension of a set direction, the movement distance of the feature in the set direction within the first time interval is calculated;

[0130] Based on the distance traveled and the state prediction and motion state information within the first time interval, the motion smoothness of the feature in the position dimension of the set direction is evaluated.

[0131] In some embodiments of this application, the video smoothness evaluation apparatus further includes an initial evaluation module and a target evaluation module, wherein,

[0132] The initial evaluation module is used to determine the initial evaluation model for the features;

[0133] The target evaluation module is used to optimize the initial evaluation model based on the motion state information set to obtain a target evaluation model with features.

[0134] In some embodiments of this application, the initial evaluation model includes multiple initial parameters, and the target evaluation module can be specifically used for:

[0135] Input multiple time points and multiple motion state information into the initial evaluation model to obtain the first output value;

[0136] If the first output value does not match the preset output value, or the number of parameter updates of the initial evaluation model is less than the preset number, update the initial parameters to obtain an updated evaluation model, determine the updated evaluation model as the new initial evaluation model, and return to execute the step of inputting multiple time moments and multiple motion state information into the initial evaluation model to obtain the first output value;

[0137] If the first output value matches the preset output value, or the number of parameter updates of the initial evaluation model is equal to the preset number, the initial evaluation model is determined as the target evaluation model with the following features.

[0138] In some embodiments of this application, the target video includes a video uploaded by a user on a streaming media platform, and the video smoothness evaluation device further includes a frame rate adjustment module, which is specifically used for:

[0139] If the video smoothness is greater than the first preset smoothness, reduce the frame rate of the target video to obtain the first processed video;

[0140] If the video smoothness is lower than the second preset smoothness and the target video carries a hotspot video identifier, the frame rate of the target video is increased to obtain the second processed video.

[0141] In some embodiments of this application, the target video includes a video generated by a generative video platform based on a first video generation algorithm. The generative video platform also includes at least one second video generation algorithm. The video smoothness evaluation device further includes a generation adjustment module, which is specifically used for:

[0142] If the video smoothness is less than the third preset smoothness corresponding to the first video generation algorithm, the generative video platform is triggered to regenerate the target video based on the second video generation algorithm.

[0143] This application embodiment can acquire a target video, which includes multiple video frames. Feature extraction is performed on the video frames to obtain multiple features and motion state information of the features. The matching relationship between features in adjacent video frames is calculated to obtain a motion state information set for each feature. This motion state information set includes the motion state information of the feature in different video frames. Smoothness evaluation is performed on the features based on the motion state information set to obtain the motion smoothness of each feature. Multiple motion smoothness values ​​are aggregated to obtain the video smoothness of the target video. This application can perform smoothness calculation at the feature level in a more refined manner, and then aggregate the motion smoothness values ​​of all features to obtain a more accurate video smoothness of the target video.

[0144] This application also provides a computer device, such as... Figure 10As shown, it illustrates a structural schematic diagram of a computer device involved in an embodiment of this application. This computer device can be a terminal or a server, specifically:

[0145] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 10 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0146] The processor 401 is the control center of the computer device, connecting various parts of the computer device through various interfaces and lines. It performs various functions and processes data by running or executing computer programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and computer programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0147] The memory 402 can be used to store computer programs and modules. The processor 401 executes various functional applications and data processing by running the computer programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, computer programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0148] The computer device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0149] The computer device may also include an input unit 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0150] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the computer device loads the executable files corresponding to the processes of one or more computer programs into the memory 402 according to the following instructions, and the processor 401 runs the application programs stored in the memory 402 to realize various functions, as follows:

[0151] The target video is acquired, which includes multiple video frames. Feature extraction is performed on the video frames to obtain multiple features of the video frames and the motion state information of the features. The matching relationship of features in adjacent video frames is calculated to obtain the motion state information set of each feature. The motion state information set includes the motion state information of the feature in different video frames. The smoothness of the features is evaluated based on the motion state information set to obtain the motion smoothness of each feature. The multiple motion smoothness values ​​are aggregated to obtain the video smoothness of the target video.

[0152] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0153] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0154] Therefore, embodiments of this application provide a computer-readable storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the video smoothness evaluation methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0155] The target video is acquired, which includes multiple video frames. Feature extraction is performed on the video frames to obtain multiple features of the video frames and the motion state information of the features. The matching relationship of features in adjacent video frames is calculated to obtain the motion state information set of each feature. The motion state information set includes the motion state information of the feature in different video frames. The smoothness of the features is evaluated based on the motion state information set to obtain the motion smoothness of each feature. The multiple motion smoothness values ​​are aggregated to obtain the video smoothness of the target video.

[0156] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0157] Since the computer program stored in the computer-readable storage medium can execute the steps in any of the video smoothness evaluation methods provided in the embodiments of this application, the beneficial effects that any of the video smoothness evaluation methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0158] This application also provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods provided in various optional implementations of the video smoothness evaluation method described above.

[0159] The above provides a detailed description of a video smoothness evaluation method, apparatus, and computer device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for evaluating video smoothness, characterized in that, include: Acquire a target video, which includes multiple video frames; Feature extraction is performed on the video frame to obtain multiple features of the video frame and motion state information of the features; Calculate the matching relationship of features in adjacent video frames to obtain a motion state information set for each feature, wherein the motion state information set includes the motion state information of the feature in different video frames; The smoothness of the features is evaluated based on the motion state information set to obtain the motion smoothness of each feature. The video smoothness of the target video is obtained by aggregating multiple motion smoothness metrics.

2. The method according to claim 1, characterized in that, The plurality of video frames correspond to different times in the target video, and the motion state information set includes the motion state information of the feature at different times. The step of evaluating the feature smoothness based on the motion state information set to obtain the motion smoothness of each feature includes: The state prediction information at different times is determined through the target evaluation model; Based on the state prediction information and the corresponding motion state information at the time, the motion smoothness of the feature is evaluated.

3. The method according to claim 2, characterized in that, The moment in the motion state information set corresponds to at least two different dimensions of motion state information, and each dimension corresponds to a target evaluation model. The evaluation of the motion smoothness of the feature based on the state prediction information and the corresponding moment's motion state information includes: Based on the state prediction information and the corresponding motion state information at different times, the motion smoothness of the feature in different dimensions is evaluated. The motion smoothness of the feature is obtained by aggregating the motion smoothness of at least two different dimensions.

4. The method according to claim 3, characterized in that, The dimension includes a position dimension with a defined direction. The evaluation of the motion smoothness of the feature across different dimensions, based on the state prediction information and its corresponding motion state information at different times, includes: The movement distance of the feature in the set direction is calculated within the first time interval by using the motion state information at different times under the position dimension of the set direction; Based on the travel distance, the state prediction information and the motion state information within the first time interval, the motion smoothness of the feature in the position dimension of the set direction is evaluated.

5. The method according to claim 2, characterized in that, The method further includes: Determine an initial evaluation model for the aforementioned features; The initial evaluation model is optimized based on the motion state information set to obtain the target evaluation model for the feature.

6. The method according to claim 5, characterized in that, The initial evaluation model includes multiple initial parameters. Optimizing the initial evaluation model based on the motion state information set to obtain the target evaluation model for the features includes: The initial evaluation model is input with multiple time points and multiple motion state information to obtain a first output value; If the first output value does not match the preset output value, or the number of parameter updates of the initial evaluation model is less than the preset number, the initial parameters are updated to obtain an updated evaluation model. The updated evaluation model is determined to be the new initial evaluation model, and the process returns to the step of inputting multiple times and multiple motion state information into the initial evaluation model to obtain the first output value. If the first output value matches the preset output value, or the number of parameter updates of the initial evaluation model is equal to the preset number, the initial evaluation model is determined to be the target evaluation model for the feature.

7. The method according to any one of claims 1-6, characterized in that, The target video includes videos uploaded by users on streaming media platforms, and the method further includes: If the video smoothness is higher than the first preset smoothness, the frame rate of the target video is reduced to obtain the first processed video; If the smoothness of the video is lower than the second preset smoothness and the target video carries a hotspot video identifier, the frame rate of the target video is increased to obtain the second processed video.

8. The method according to any one of claims 1-6, characterized in that, The target video includes videos generated by a generative video platform based on a first video generation algorithm, the generative video platform further includes at least one second video generation algorithm, and the method further includes: If the video smoothness is less than the third preset smoothness corresponding to the first video generation algorithm, the generative video platform is triggered to regenerate the target video based on the second video generation algorithm.

9. A video fluency evaluation device, characterized in that, include: The acquisition module is used to acquire a target video, which includes multiple video frames; The feature module is used to extract features from the video frame to obtain multiple features of the video frame and motion state information of the features; The calculation module is used to calculate the matching relationship of features in adjacent video frames to obtain a motion state information set for each feature, wherein the motion state information set includes the motion state information of the feature in different video frames; The evaluation module is used to evaluate the smoothness of the features based on the motion state information set, and obtain the motion smoothness of each feature; An aggregation module is used to aggregate multiple motion smoothness metrics to obtain the video smoothness of the target video.

10. A computer device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor is used to run the application program within the memory to perform the steps in the video smoothness evaluation method according to any one of claims 1 to 8.

Images