Training method, evaluation method and device of evaluation model of video fluency, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a training method, an evaluation method and a device of an evaluation model of video fluency, electronic equipment, a storage medium and a computer program product, and the training method comprises the steps: obtaining a video sample which is provided with a fluency label; inputting a video frame of the video sample into a visual encoder to obtain a plurality of space-time visual features; inputting the plurality of space-time visual features into a space-time visual feature compressor for compression to obtain a compressed space-time visual feature with a preset length; inputting the compressed space-time visual features with a preset length into an evaluation model to obtain the fluency of the video sample predicted by the evaluation model; calculating loss based on the predicted fluency of the video sample and the fluency label of the video sample; parameters of the spatio-temporal visual feature compressor and the evaluation model are adjusted based on the loss.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video, and more particularly, to a training method and an evaluation method of an evaluation model for video fluency, an evaluation device, an electronic device, a storage medium, and a computer program product. BACKGROUND

[0002] There are many factors affecting the user video Quality of Experience (QoE), such as real-time performance, fluency, definition, first-frame image output, and audio-video synchronization, etc. Among them, the fluency of the video usually refers to the performance of the picture continuity and stability during the video playback, which is mainly used to measure the smooth procedure of the video playback and whether there are problems such as freezing and frame skipping. Compared with other factors, users are more sensitive to the change of fluency, because video freezing and delay will directly interrupt the user's viewing fluency, and seriously affect the user experience. Therefore, it is of great significance to accurately evaluate the fluency of the video.

[0003] In the related art, the model is mainly used to evaluate the fluency of the video. However, the model in the related art only supports fixed-length video input. In order to train the model using videos of different lengths, frame extraction processing needs to be performed on the video. However, frame extraction of the video will lose the video timing information, making it difficult for the model to accurately learn the fluency characteristics between the video frames, and thus resulting in poor training effect of the model, which makes it difficult to accurately evaluate the fluency of the video. SUMMARY

[0004] The present disclosure provides a training method and an evaluation method of an evaluation model for video fluency, an evaluation device, an electronic device, a storage medium, and a computer program product to at least solve the problem that the model has poor training effect and is difficult to accurately evaluate the fluency of the video due to the need for frame extraction processing on the video in advance in the related art.

[0005] According to a first aspect of an embodiment of the present disclosure, a training method of an evaluation model for video fluency is provided, comprising: obtaining a video sample, wherein the video sample has a fluency label; inputting a video frame of the video sample into a visual encoder to obtain a plurality of spatio-temporal visual features; inputting the plurality of spatio-temporal visual features into a spatio-temporal visual feature compressor for compression to obtain compressed spatio-temporal visual features of a preset length; inputting the compressed spatio-temporal visual features of the preset length into the evaluation model to obtain the fluency of the video sample predicted by the evaluation model; calculating a loss based on the predicted fluency of the video sample and the fluency label of the video sample; and adjusting parameters of the spatio-temporal visual feature compressor and the evaluation model based on the loss.

[0006] Optionally, the obtaining the video samples comprises: obtaining an original video sample and a plurality of frame loss video samples corresponding to the original video sample, wherein the original video sample has an absolute smoothness label used to indicate a real smoothness of the original video sample; and wherein the obtaining the plurality of frame loss video samples comprises: performing frame loss processing on the original video sample according to each of a plurality of preset frame loss rates to obtain the plurality of frame loss video samples, wherein any two video samples of the plurality of frame loss video samples and the original video sample have a relative smoothness label used to indicate a high-low contrast relationship of real smoothness corresponding to the any two video samples respectively; and the calculating the loss based on the predicted smoothness of the video samples and the smoothness labels corresponding to the video samples comprises: calculating the loss based on the predicted smoothness of the original video sample, the absolute smoothness label, the relative smoothness labels corresponding to the any two video samples, and the predicted smoothness of each of the any two video samples.

[0007] Optionally, the calculating the loss based on the predicted smoothness of the original video sample, the absolute smoothness label, the relative smoothness labels corresponding to the any two video samples, and the predicted smoothness of each of the any two video samples comprises: calculating a first loss based on the predicted smoothness of the original video sample and the absolute smoothness label; and calculating a second loss based on the relative smoothness labels corresponding to the any two video samples and the predicted smoothness of each of the any two video samples; and the adjusting the parameters of the spatio-temporal visual feature compressor and the evaluation model based on the loss comprises: adjusting the parameters of the spatio-temporal visual feature compressor and the evaluation model based on the first loss and the second loss.

[0008] Optionally, the obtaining the plurality of frame loss video samples comprises: for each of the plurality of preset frame loss rates, calculating a frame loss number corresponding to the preset frame loss rate based on the preset frame loss rate and a total number of video frames contained in the original video sample; dividing the original video sample into T time intervals; and according to the frame loss number corresponding to the preset frame loss rate, randomly selecting at least one video frame from a plurality of video frames in each of the T time intervals to discard to obtain a frame loss video sample corresponding to the preset frame loss rate.

[0009] Optionally, the first loss is a regression loss and the second loss is a ranking loss.

[0010] Optionally, the spatio-temporal visual feature compressor comprises an attention network and a feedforward neural network, the attention network corresponds to a learnable feature representation matrix; the inputting the plurality of spatio-temporal visual features into the spatio-temporal visual feature compressor for compression to obtain the compressed spatio-temporal visual features of the preset length comprises: inputting the plurality of spatio-temporal visual features into the attention network to obtain an attention result; superimposing the attention result and the learnable feature representation matrix to obtain a first superimposed result; inputting the first superimposed result into the feedforward neural network to obtain a feedforward output result; and superimposing the feedforward output result and the first superimposed result to obtain the compressed spatio-temporal visual features of the preset length.

[0011] Optionally, the absolute fluency label corresponding to the original video sample is an absolute fluency label obtained by a user scoring the original video sample based on a preset fluency scoring benchmark, wherein the preset fluency scoring benchmark comprises a plurality of preset scoring levels, and the higher the preset scoring level is, the better the video fluency indicated by the preset scoring level is.

[0012] Optionally, the video samples used for training the evaluation model cover a plurality of factors affecting video fluency; the plurality of factors comprise at least two of the following: video shooting conditions, video shooting subjects, video shooting backgrounds, video shooting devices, video resolutions, and video shooting scenes.

[0013] Optionally, before the inputting the video frames of the video sample into the visual encoder to obtain the plurality of spatio-temporal visual features, the training method further comprises: performing continuous segment sampling on the video sample to obtain a continuous video segment sampled from the video sample; and the inputting the video frames of the video sample into the visual encoder to obtain the plurality of spatio-temporal visual features comprises: inputting the video frames contained in the continuous video segment into the visual encoder to obtain the plurality of spatio-temporal visual features.

[0014] According to a second aspect of the embodiments of the present disclosure, a video fluency evaluation method is provided, comprising: obtaining a target video to be evaluated; inputting target video frames contained in the target video to be evaluated into a visual encoder to obtain a plurality of target spatio-temporal visual features; inputting the plurality of target spatio-temporal visual features into a spatio-temporal visual feature compressor trained according to the training method of the present disclosure for compression to obtain target compressed spatio-temporal visual features of a preset length; and inputting the target compressed spatio-temporal visual features of the preset length into an evaluation model trained according to the training method of the present disclosure to predict the fluency of the target video to be evaluated by the trained evaluation model.

[0015] According to a third aspect of the embodiments of the present disclosure, a device for training a video smoothness evaluation model is provided, comprising: a sample obtaining module configured to obtain a video sample, wherein the video sample has a smoothness label; an encoding module configured to input a video frame of the video sample into a visual encoder to obtain a plurality of spatio-temporal visual features; a compression module configured to input the plurality of spatio-temporal visual features into a spatio-temporal visual feature compressor to compress the plurality of spatio-temporal visual features to obtain compressed spatio-temporal visual features of a preset length; a smoothness prediction module configured to input the compressed spatio-temporal visual features of the preset length into the evaluation model to obtain a predicted smoothness of the video sample by the evaluation model; a loss calculation module configured to calculate a loss based on the predicted smoothness of the video sample and the smoothness label of the video sample; and a parameter adjustment module configured to adjust parameters of the spatio-temporal visual feature compressor and the evaluation model based on the loss.

[0016] Optionally, the sample obtaining module is configured to obtain an original video sample and a plurality of frame loss video samples corresponding to the original video sample, wherein the original video sample has an absolute smoothness label, and the absolute smoothness label is used to indicate a real smoothness of the original video sample; wherein the sample obtaining module is specifically configured to perform frame loss processing on the original video sample according to each of a plurality of preset frame loss rates to obtain the plurality of frame loss video samples, wherein any two video samples in the plurality of frame loss video samples and the original video sample have a relative smoothness label, and the relative smoothness label is used to indicate a high-low contrast relationship of real smoothness corresponding to the any two video samples respectively; and the loss calculation module is configured to calculate the loss based on a predicted smoothness corresponding to the original video sample, the absolute smoothness label, relative smoothness labels corresponding to the any two video samples, and a predicted smoothness corresponding to each of the any two video samples.

[0017] Optionally, the loss calculation module is configured to calculate a first loss based on the predicted smoothness corresponding to the original video sample and the absolute smoothness label, and calculate a second loss based on the relative smoothness labels corresponding to the any two video samples and the predicted smoothness corresponding to each of the any two video samples; and the parameter adjustment module is configured to adjust the parameters of the spatio-temporal visual feature compressor and the evaluation model based on the first loss and the second loss.

[0018] Optionally, the sample obtaining module is specifically configured to: for each preset frame loss rate, calculate a frame loss number corresponding to the preset frame loss rate based on the preset frame loss rate and a total number of video frames contained in the original video sample; divide the original video sample into T time intervals; and randomly select at least one video frame from a plurality of video frames in each time interval of the T time intervals to discard according to the frame loss number corresponding to the preset frame loss rate, to obtain a frame loss video sample corresponding to the preset frame loss rate.

[0019] Optionally, the first loss is a regression loss, and the second loss is a ranking loss.

[0020] Optionally, the spatio-temporal visual feature compressor comprises an attention network and a feed-forward neural network, the attention network corresponds to a learnable feature representation matrix; the compression module is configured to: input the plurality of spatio-temporal visual features into the attention network to obtain an attention result; superimpose the attention result and the learnable feature representation matrix to obtain a first superimposed result; input the first superimposed result into the feed-forward neural network to obtain a feed-forward output result; and superimpose the feed-forward output result and the first superimposed result to obtain the compressed spatio-temporal visual feature of the preset length.

[0021] Optionally, the absolute fluency label corresponding to the original video sample is an absolute fluency label obtained by a user performing fluency scoring on the original video sample based on a preset fluency scoring benchmark, wherein the preset fluency scoring benchmark comprises a plurality of preset scoring levels, and the higher the preset scoring level is, the better the video fluency indicated by the preset scoring level is.

[0022] Optionally, the video samples used for training the evaluation model cover a plurality of factors affecting video fluency; the plurality of factors comprise at least two of the following: video shooting condition, video shooting subject, video shooting background, video shooting device, video resolution, and video shooting scene.

[0023] Optionally, the training device further comprises a sampling module configured to sample continuous video clips from the video sample, to obtain the continuous video clips sampled from the video sample; and the encoding module is configured to input video frames contained in the continuous video clips into the visual encoder to obtain the plurality of spatio-temporal visual features.

[0024] According to a fourth aspect of the embodiments of the present disclosure, an evaluation device for video fluency is provided, comprising: a target video acquisition module configured to acquire a target video to be evaluated; a target feature acquisition module configured to input target video frames contained in the target video to be evaluated into a visual encoder to obtain a plurality of target spatio-temporal visual features; a target compressed feature acquisition module configured to input the plurality of target spatio-temporal visual features into a spatio-temporal visual feature compressor trained according to the training method of the present disclosure to compress, and obtain target compressed spatio-temporal visual features of a preset length; and a fluency evaluation module configured to input the target compressed spatio-temporal visual features of the preset length into an evaluation model trained according to the training method of the present disclosure, to predict the fluency of the target video to be evaluated by the trained evaluation model.

[0025] According to a fifth aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the training method of the evaluation model for video fluency or the evaluation method for video fluency according to the present disclosure.

[0026] According to a sixth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the training method of the evaluation model for video fluency or the evaluation method for video fluency according to the present disclosure.

[0027] According to a seventh aspect of the embodiments of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the training method of the evaluation model for video fluency or the evaluation method for video fluency according to the present disclosure.

[0028] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects: In the present disclosure, by setting a spatio-temporal visual feature compressor as a pre-module before the evaluation model for video fluency, the spatio-temporal visual feature compressor can be used to change the indefinite length of the video token input into a fixed length of the video token input. In this way, the evaluation model can be directly trained using multiple complete videos of indefinite length, i.e., multiple complete videos of non-fixed length, without the need for frame extraction processing of videos of various lengths. Since the frame extraction operation is not required before model training, the loss of video timing information can be avoided, i.e., the loss of video fluency information can be avoided, and the continuity of the video can be protected from being destroyed. In this way, the model can accurately learn the fluency characteristics between video frames, thereby effectively improving the training effect and ensuring the accuracy of the fluency evaluation of the trained model.

[0029] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory and are not restrictive of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and serve to explain the principles of the disclosure, and do not limit the disclosure.

[0031] Figure 1 is a flowchart illustrating a training method of an evaluation model of video smoothness according to an example embodiment of the disclosure; Figure 2 is a schematic diagram illustrating original video samples and frame-dropped video samples according to an example embodiment of the disclosure; Figure 3 is a schematic diagram illustrating uniform sampling and continuous segment sampling according to an example embodiment of the disclosure; Figure 4 is a schematic diagram illustrating a visual encoder and a spatio-temporal visual feature compressor according to an example embodiment of the disclosure; Figure 5 is a scatter plot illustrating a smoothness objective index in the related art and a subjective score of video smoothness provided by the disclosure; Figure 6 is a flowchart illustrating an evaluation method of video smoothness according to an example embodiment of the disclosure; Figure 7 is a block diagram illustrating a training device of an evaluation model of video smoothness according to an example embodiment of the disclosure; Figure 8 is a block diagram illustrating an evaluation device of video smoothness according to an example embodiment of the disclosure; Figure 9 is a block diagram illustrating an electronic device according to an example embodiment of the disclosure. DETAILED DESCRIPTION

[0032] In order for those skilled in the art to better understand the technical solutions of the disclosure, the technical solutions in the embodiments of the disclosure will be described clearly and completely below with reference to the drawings.

[0033] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present disclosure and the above-described drawings are used to distinguish similar objects and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] It should be noted herein that "at least one of a plurality" appearing in the present disclosure means that it includes "any one of the plurality", "a combination of any multiple of the plurality", and "all of the plurality" in the three types of alternatives. For example, "including at least one of A and B" includes the following three alternatives: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step one and step two" means the following three alternatives: (1) performing step one; (2) performing step two; (3) performing step one and step two.

[0035] It is of great significance to accurately assess the smoothness of a video. First, accurate smoothness assessment helps improve user QoE, reduces discomfort caused by stuttering and delay, and thus increases user viewing time and frequency. Second, accurate smoothness assessment helps improve platform service quality, helping video service providers identify and solve bottlenecks in network transmission or decoding process to ensure more stable service. In addition, accurate smoothness assessment also helps save platform costs, helping operators better manage and allocate network bandwidth, and reduce congestion and data packet loss.

[0036] Currently, video smoothness assessment has not been studied as a separate task, but smoothness is considered as a sub-dimension of the Video Quality Assessment (VQA) task. However, this task paradigm has corresponding limitations. First, current VQA methods focus more on the clarity of video quality, while users are usually more sensitive to changes in smoothness than changes in clarity, because stuttering and delay will directly interrupt the continuity of viewing. Second, video service providers often need to balance clarity, smoothness, and cost in order to improve user QoE while balancing Return on Investment (ROI), which emphasizes the need to decouple and evaluate smoothness from the VQA task.

[0037] As described above, in the related art, the model is mainly used to evaluate the fluency of the video. However, the model in the related art only supports fixed-length video input, that is, it only accepts fixed-length video input. In order to train the model using videos of different lengths, frame extraction needs to be performed on the video, that is, a fixed number of video frames need to be extracted from the video to input the model for training. However, frame extraction of the video will lose the video timing information, that is, the fluency information of the video will be lost, which will destroy the continuity of the video, making it difficult for the model to accurately learn the fluency characteristics between the video frames, and thus resulting in poor training effect of the model, making it difficult to accurately evaluate the fluency of the video.

[0038] In addition, the objective indicators for evaluating the fluency of the video in the related art mainly include frame rate, stutter rate, jank number and the like, but these objective indicators are difficult to align with the real user experience quality, resulting in that they cannot be used as reliable indicators for measuring fluency.

[0039] In order to solve the above problems in the related art, the training method, the evaluation method, the device, the electronic equipment, the storage medium and the computer program product of the evaluation model of the video fluency provided by the present disclosure can realize the conversion of the indefinite-length video token input into the fixed-length video token input by setting the spatio-temporal visual feature compressor as a pre-module before the evaluation model of the video fluency. In this way, the indefinite-length multiple complete videos, that is, the multiple complete videos of non-fixed length can be directly used to train the evaluation model, and frame extraction of videos of various lengths is not required for training. Since the frame extraction operation is not required before the model training, the video timing information can be avoided to be lost, that is., the fluency information of the video can be avoided to be lost, and the continuity of the video can be protected from being destroyed. In this way, the model can accurately learn the fluency characteristics between the video frames, and thus the training effect of the model can be effectively improved, and the accuracy of the fluency evaluation of the model trained can be ensured.

[0040] Figure 1 is a flowchart illustrating a training method of an evaluation model of video fluency according to an example embodiment of the present disclosure.

[0041] Referring to Figure 1 In step 101, a video sample can be obtained, wherein the video sample can have a fluency label.

[0042] According to an example embodiment of the present disclosure, the video samples used to train the evaluation model can cover multiple factors affecting video smoothness, and the multiple factors can include at least two of the following: video shooting conditions, video shooting subjects, video shooting backgrounds, video shooting devices, video resolutions, and video shooting scenes. For example, the present disclosure samples 10K videos covering the above-mentioned multiple factors affecting video smoothness from the six data sets of SSv2, Maxwell, LSVQ, KonViD-1k, LIVE-VQC, and YouTube-UGC. Moreover, in order to exclude the interference of low-quality videos on the smoothness score, the present disclosure filters out videos with a Mean Opinion Score (MOS) lower than 3.0, and finally obtains 4605 benchmark videos used to train the evaluation model.

[0043] In this way, by training the evaluation model using video samples covering multiple factors affecting video smoothness, the trained evaluation model can cope with various types of videos, thereby ensuring the generalization ability and robustness of the evaluation model.

[0044] In related technologies, commonly used VQA data sets use manual scoring methods. Since the number of video samples included in the data set is generally large, the manual scoring method consumes high labor costs and time costs. In view of this, the present disclosure proposes a method of automatically generating homologous videos with different freezing degrees by controlling the frame loss rate.

[0045] According to an example embodiment of the present disclosure, an original video sample and multiple frame loss video samples corresponding to the original video sample can be obtained. The original video sample can have an absolute smoothness label, which can be used to indicate the true smoothness of the original video sample.

[0046] In addition, the original video sample can be subjected to frame loss processing according to each of multiple preset frame loss rates, and thus multiple frame loss video samples can be obtained. Moreover, any two video samples of the multiple frame loss video samples and the original video sample can have a relative smoothness label, which can be used to indicate the high-low contrast relationship of the true smoothness corresponding to any two video samples, respectively.

[0047] Next, the loss of the evaluation model can be calculated based on the predicted smoothness corresponding to the original video sample, the absolute smoothness label, the relative smoothness label corresponding to any two video samples, and the predicted smoothness corresponding to each video sample of any two video samples.

[0048] Thus, since the original video sample can be frame-dropped according to multiple preset frame-dropping rates respectively, and the greater the frame-dropping rate is, the worse the fluency of the corresponding frame-dropped video sample will be. Therefore, this way of automatically generating homologous videos with different degrees of stuttering is equivalent to automatically labeling different frame-dropped video samples with corresponding fluency labels. Compared with the manual labeling method, the automatic labeling method provided by the present disclosure can effectively save labor costs and time costs, thereby significantly improving the efficiency of labeling.

[0049] According to an exemplary embodiment of the present disclosure, for each preset frame-dropping rate, the number of frames dropped corresponding to the preset frame-dropping rate can be calculated based on the preset frame-dropping rate and the total number of video frames contained in the original video sample. Then, the original video sample can be divided into T time intervals. Next, at least one video frame can be randomly selected from the multiple video frames in each time interval of the T time intervals for discarding according to the number of frames dropped corresponding to the preset frame-dropping rate, and thus the frame-dropped video sample corresponding to the preset frame-dropping rate can be obtained.

[0050] Figure 2 is a schematic diagram showing the original video sample and the frame-dropped video sample according to an exemplary embodiment of the present disclosure. Referring to Figure 2 , a total of 3 types of video samples are shown, and each row corresponds to the original video sample and the frame-dropped video sample of one type of content. Moreover, the leftmost side of each row is the original video sample, and the video samples other than the original video sample in each row are frame-dropped video samples. For example, the aforementioned multiple preset frame-dropping rates can be: drop rate = 0.2, drop rate = 0.5, and drop rate = 0.8. In addition, the number of frames dropped corresponding to each preset frame-dropping rate can be allocated to the T time intervals contained in the original video sample, and the number of discarded video frames allocated to each time interval is random, so that the stuttering video in the real world can be maximally approximated.

[0051] It should be noted that, as mentioned above, the greater the frame-dropping rate is, the worse the fluency of the corresponding frame-dropped video sample will be. Therefore, Figure 2 The video sample of each type of content in (original video sample, drop rate = 0.2 of the frame loss video sample), (original video sample, drop rate = 0.5 of the frame loss video sample), (original video sample, drop rate = 0.8 of the frame loss video sample), (drop rate = 0.2 of the frame loss video sample, drop rate = 0.5 of the frame loss video sample), (drop rate = 0.2 of the frame loss video sample, drop rate = 0.8 of the frame loss video sample), (drop rate = 0.5 of the frame loss video sample, drop rate = 0.8 of the frame loss video sample).

[0052] In addition, according to the principle that the greater the frame loss rate is, the worse the fluency of the corresponding frame loss video sample is, each video sample can correspond to a relative fluency label. For example, for the video sample pair (original video sample, drop rate = 0.2 of the frame loss video sample), the relative fluency label can be: “the real fluency of the original video sample is higher than the real fluency of the drop rate = 0.2 of the frame loss video sample”; or for the video sample pair (drop rate = 0.2 of the frame loss video sample, drop rate = 0.8 of the frame loss video sample), the relative fluency label can be: “the real fluency of the drop rate = 0.2 of the frame loss video sample is higher than the real fluency of the drop rate = 0.8 of the frame loss video sample”, and the like.

[0053] In step 102, the video frames of the video sample can be input into a visual encoder to obtain a plurality of spatio-temporal visual features.

[0054] It should be noted that the manner of sampling the video usually includes uniform sampling, frame rate sampling (FPS sampling) and continuous segment sampling (dense sampling). Figure 3 is a schematic diagram showing uniform sampling and continuous segment sampling according to an example embodiment of the present disclosure. Referring to Figure 3 , uniform sampling can refer to sampling once every fixed number of video frames, in Figure 3 , it is sampling once every 3 video frames. Continuous segment sampling refers to directly taking a certain continuous segment of the original video as the sampling result.

[0055] Uniform sampling and frame rate sampling achieve better results than continuous segment sampling in video understanding tasks, because the former two sampling methods can cover as much semantic information of the whole video as possible, while continuous segment sampling is easy to make the model focus on only part of the video, resulting in the loss of global information. In the present disclosure, the video samples can be sampled using the continuous segment sampling method in the pre-processing stage. This is because the video fluency evaluation task is different from the semantic level video understanding task, and the video fluency evaluation task requires the model to focus on the subtle changes between frames. The continuous segment sampling method can preserve the continuity information between video frames, so that the evaluation model can better learn the fluency characteristics between video frames.

[0056] According to an exemplary embodiment of the present disclosure, the video samples can be continuously sampled to obtain continuous video segments sampled from the video samples. Next, the video frames contained in the continuous video segments obtained by sampling can be input into a vision encoder to obtain a plurality of spatio-temporal vision features (tokens).

[0057] Figure 4 is a schematic diagram showing a vision encoder and a spatio-temporal vision feature compressor according to an exemplary embodiment of the present disclosure. Referring to Figure 4 The three video frames sampled, i.e. the video frame at t=0, the video frame at t=1 and the video frame at t=2, can be input into the vision encoder respectively, and each vision encoder can output a plurality of spatio-temporal vision features corresponding to the respective video frame. For example, each vision encoder can output 3 3=9 spatio-temporal vision features, wherein each spatio-temporal vision feature is used to represent the visual characteristics at a corresponding position in the video frame.

[0058] In step 103, the aforementioned plurality of spatio-temporal vision features can be input into the spatio-temporal vision feature compressor for compression to obtain compressed spatio-temporal vision features of a preset length. For example, as described above, Figure 4 Each video frame in the video frame sequence can correspond to 3 3=9 spatio-temporal vision features, so the three video frames can correspond to 3 9=27 spatio-temporal vision features in total. The 27 spatio-temporal vision features can be flattened and spliced to obtain spliced spatio-temporal vision features: .

[0059] For example, the aforementioned "preset length" can be, but is not limited to, 5, i.e. no matter how many and how long the spatio-temporal vision features input into the spatio-temporal vision feature compressor are, the output of the spatio-temporal vision feature compressor is fixed to 5 compressed spatio-temporal vision features.

[0060] According to an example embodiment of the present disclosure, the aforementioned spatio-temporal visual feature compressor can comprise an attention network and a feed forward neural network (FFW), and the attention network can correspond to a learnable feature representation matrix X.

[0061] The aforementioned plurality of spatio-temporal visual features can be input into the attention network to obtain an attention result. For example, the attention result can be calculated by the following formula: Y= wherein Y is the attention result, X is a learnable feature representation matrix corresponding to the attention network, is the aforementioned spliced spatio-temporal visual feature, is the dimension of the spatio-temporal visual feature, for example, may be but is not limited to 768.

[0062] Then, the attention result can be superimposed with the learnable feature representation matrix to obtain a first superimposed result. Next, the first superimposed result can be input into the feed forward neural network to obtain a feed forward output result.

[0063] Then, the feed forward output result can be superimposed with the first superimposed result to obtain a compressed spatio-temporal visual feature of a preset length.

[0064] In this way, in the present disclosure, the spatio-temporal visual feature compressor, i.e., the token compression structure, can map the spatio-temporal visual features output by the visual encoder into a fixed number of output tokens, i.e., it can realize the conversion of indefinite-length video token input into definite-length video token input using the spatio-temporal visual feature compressor. It is irrelevant to the resolution of the input image and the number of input video frames, and can ensure that the evaluation model can accept video inputs of different lengths and can guarantee that the smoothness change information between video frames is not lost, so that the model can accurately learn the smoothness characteristics between video frames, ensuring good smoothness perception ability. Further, by compressing the spatio-temporal visual features output by the visual encoder through the spatio-temporal visual feature compressor, the problem of computational explosion caused by continuous segment sampling can also be effectively solved, i.e., the operation efficiency can be effectively improved.

[0065] In step 104, the compressed spatio-temporal visual feature of the preset length can be input into the evaluation model to obtain the smoothness of the video sample predicted by the evaluation model.

[0066] In step 105, the loss can be calculated based on the predicted smoothness of the video sample and the smoothness label of the video sample.

[0067] According to the example embodiments of the present disclosure, the first loss can be calculated based on the predicted smoothness corresponding to the original video sample and the absolute smoothness label. The second loss can also be calculated based on the relative smoothness label corresponding to any two video samples and the predicted smoothness corresponding to each of the any two video samples. Next, the parameters of the spatio-temporal visual feature compressor and the evaluation model can be adjusted based on the first loss and the second loss.

[0068] According to the example embodiments of the present disclosure, the first loss can be a regression loss and the second loss can be a ranking loss. The ranking loss can be expressed by the following formula, for example: .

[0069] wherein, represents the predicted smoothness corresponding to the frame loss video sample with a larger frame loss rate, represents the predicted smoothness corresponding to the frame loss video sample with a smaller frame loss rate. In this way, when is less than , it indicates that the prediction of the evaluation model is accurate, and at this time, = 0, indicating that no ranking loss will be generated since the evaluation model makes an accurate prediction at this time; when is greater than , it indicates that the prediction of the evaluation model is incorrect, and at this time, = , indicating that a ranking loss will be generated since the evaluation model makes an incorrect prediction at this time.

[0070] In this way, the present disclosure provides a degradation manner of generating a stuttering video and a ranking-based training strategy matched therewith, i.e., the present disclosure provides a training strategy based on smoothness ranking of homologous video, which can achieve a large degree of expansion of the smoothness training data, thereby improving the prediction accuracy of the evaluation model.

[0071] In step 106, the parameters of the spatio-temporal visual feature compressor and the evaluation model can be adjusted based on the calculated loss to train the evaluation model.

[0072] According to the example embodiments of the present disclosure, the absolute smoothness label corresponding to the original video sample can be an absolute smoothness label obtained by a user scoring the smoothness of the original video sample based on a preset smoothness scoring benchmark. The preset smoothness scoring benchmark can include a plurality of preset scoring levels, and the higher the preset scoring level, the better the video smoothness indicated by the preset scoring level.

[0073] For example, to align the fluency evaluation benchmark with human subjective perception, by focusing on the continuity, frame rate stability and motion fluency of a video in the time dimension, the disclosure designs a five-point absolute categorical rating (ACR) scale for video temporal smoothness, and the five grades of subjective fluency score can be as follows: Level 1: Bad - The video temporal is extremely unsmooth, with frequent and obvious stuttering, frame skipping or frame rate fluctuation, making it difficult to identify the motion coherently, and causing strong discomfort when watching.

[0074] Level 2: Poor - The video temporal is obviously stuttered or unstable, and the distortion is easy to detect, although the motion can be identified, but the attention is frequently interrupted when watching.

[0075] Level 3: Fair - The video temporal is basically coherent, but there are occasional fluctuations in fluency, and the viewer needs to concentrate to detect the flaws, but the overall content understanding is not affected.

[0076] Level 4: Good - The video temporal is smooth and natural, with almost no stuttering or frame rate fluctuation, and the temporal defects are difficult to detect when watching, and the motion coherence is close to professional shooting level.

[0077] Level 5: Excellent - The video temporal is extremely coherent and smooth, with completely stable frame rate, and the motion performance is consistent with the real world perception, and the viewer is completely immersed in the content without any temporal-related attention interference.

[0078] Figure 5 is a scatter plot showing the fluency objective indicators in the related art and the video fluency subjective score provided by the disclosure. Referring to Figure 5 , the left graph is a scatter plot of the jank number in the related art and the video fluency subjective score provided by the disclosure, and the right graph is a scatter plot of the stutter rate in the related art and the video fluency subjective score provided by the disclosure.

[0079] Referring to Figure 5 the left graph, in principle, the higher the subjective score of video fluency, the fewer the jank number of the video. However, points A and B in the graph violate this principle: the subjective score of video fluency of point A is lower than that of point B, and in principle, the jank number of point A should be more than that of point B, but the graph shows that the jank number of point A is less than that of point B.

[0080] Alternatively, referring to Figure 5In the right middle graph, in principle, the higher the subjective score of video fluency, the smaller the video stutter rate should be. However, the C point and the D point in the graph violate this principle: the subjective score of video fluency of the C point video is lower than that of the D point video, and in principle, the stutter rate of the C point video should be higher than that of the D point video, but the graph shows that the stutter rate of the C point video is lower than that of the D point video.

[0081] It can be seen that the fluency objective indicators (such as the stutter rate and the number of stutters) in the related art cannot align with the subjective perception of users on video fluency. Therefore, the preset fluency score benchmark provided by the present disclosure is very necessary, which can accurately represent the subjective perception of users on video fluency, and thus can guarantee accurate recommendation of friendly fluency videos to users and can improve the viewing experience of users.

[0082] In order to solve the problems of insufficient attention to fluency in the VQA task paradigm in the related art and the difficulty of aligning the objective fluency indicators with human subjective fluency perception, the present disclosure provides a training method of a video fluency evaluation model, which makes corresponding improvements in model architecture, training data, training strategy and evaluation benchmark, and can guarantee that the trained evaluation model can accurately predict the video fluency that aligns with human subjective perception, so as to guarantee a better user viewing experience.

[0083] Figure 6 is a flowchart illustrating a video fluency evaluation method according to an example embodiment of the present disclosure.

[0084] Referring to Figure 6 In step 601, a target video to be evaluated can be obtained.

[0085] In step 602, the target video frames contained in the target video to be evaluated can be input into a visual encoder to obtain a plurality of target spatio-temporal visual features. For example, as described in the previous embodiment, the target video to be evaluated can be first sampled for continuous segments, and then the plurality of continuous video frames obtained by sampling can be input into the visual encoder respectively to obtain a plurality of target spatio-temporal visual features corresponding to each video frame.

[0086] In step 603, the plurality of target spatio-temporal visual features can be input into a spatio-temporal visual feature compressor trained according to the training method of the present disclosure for compression to obtain target compressed spatio-temporal visual features of a preset length. For example, the preset length can be, but is not limited to, 5. At this time, no matter how many and how long the spatio-temporal visual features input into the spatio-temporal visual feature compressor are, the output of the spatio-temporal visual feature compressor is fixed to 5 compressed spatio-temporal visual features.

[0087] In step 604, the preset length of the target compressed spatio-temporal visual features can be input into the evaluation model trained according to the training method of the present disclosure to predict the fluency of the target video to be evaluated by the trained evaluation model.

[0088] Figure 7 is a block diagram of a training device 700 of a video fluency evaluation model according to an example embodiment of the present disclosure.

[0089] Referring to Figure 7 The training device 700 of the video fluency evaluation model can include a sample acquisition module 701, an encoding module 702, a compression module 703, a fluency prediction module 704, a loss calculation module 705, and a parameter adjustment module 706.

[0090] The sample acquisition module 701 is configured to acquire video samples, wherein the video samples have fluency labels; The encoding module 702 is configured to input video frames of the video samples into a visual encoder to obtain a plurality of spatio-temporal visual features; The compression module 703 is configured to input the plurality of spatio-temporal visual features into a spatio-temporal visual feature compressor for compression to obtain compressed spatio-temporal visual features of a preset length; The fluency prediction module 704 is configured to input the compressed spatio-temporal visual features of the preset length into an evaluation model to obtain the fluency of the video samples predicted by the evaluation model; The loss calculation module 705 is configured to calculate a loss based on the predicted fluency of the video samples and the fluency labels of the video samples; The parameter adjustment module 706 is configured to adjust the parameters of the spatio-temporal visual feature compressor and the evaluation model based on the loss.

[0091] According to an example embodiment of the present disclosure, the sample acquisition module 701 is configured to: acquire original video samples and a plurality of frame loss video samples corresponding to the original video samples, wherein the original video samples have absolute fluency labels indicating the true fluency of the original video samples; wherein the sample acquisition module 701 is specifically configured to: frame loss processing is performed on the original video samples according to each of a plurality of preset frame loss rates to obtain a plurality of frame loss video samples, wherein any two video samples in the plurality of frame loss video samples and the original video samples have relative fluency labels indicating the high-low contrast relationship of the true fluency of any two video samples respectively; The loss calculation module 705 is configured to: The loss is calculated based on the predicted fluency corresponding to the original video sample, the absolute fluency label, the relative fluency label corresponding to any two video samples, and the predicted fluency corresponding to each video sample in any two video samples.

[0092] According to an example embodiment of the present disclosure, the loss calculation module 705 is configured to: calculate a first loss based on the predicted fluency corresponding to the original video sample and the absolute fluency label; calculate a second loss based on the relative fluency label corresponding to any two video samples and the predicted fluency corresponding to each video sample in any two video samples; The parameter adjustment module 706 is configured to: adjust the parameters of the spatio-temporal visual feature compressor and the evaluation model based on the first loss and the second loss.

[0093] According to an example embodiment of the present disclosure, the sample acquisition module 701 is specifically configured to: for each preset frame loss rate, calculate the number of frames lost corresponding to the preset frame loss rate based on the preset frame loss rate and the total number of video frames contained in the original video sample; divide the original video sample into T time intervals; discard at least one video frame from the multiple video frames in each time interval of the T time intervals according to the number of frames lost corresponding to the preset frame loss rate, to obtain a frame loss video sample corresponding to the preset frame loss rate.

[0094] According to an example embodiment of the present disclosure, the first loss is a regression loss, and the second loss is a ranking loss.

[0095] According to an example embodiment of the present disclosure, the spatio-temporal visual feature compressor comprises an attention network and a feedforward neural network, and the attention network corresponds to a learnable feature representation matrix; The compression module 703 is configured to: input the multiple spatio-temporal visual features into the attention network to obtain an attention result; superimpose the attention result and the learnable feature representation matrix to obtain a first superimposed result; input the first superimposed result into the feedforward neural network to obtain a feedforward output result; superimpose the feedforward output result and the first superimposed result to obtain compressed spatio-temporal visual features of a preset length.

[0096] According to an example embodiment of the present disclosure, the absolute fluency label corresponding to the original video sample is an absolute fluency label obtained by the user scoring the original video sample based on a preset fluency scoring benchmark, wherein the preset fluency scoring benchmark comprises a plurality of preset scoring levels, and the higher the preset scoring level, the better the video fluency indicated by the preset scoring level.

[0097] According to an example embodiment of the present disclosure, the video samples used to train the evaluation model cover a plurality of factors affecting video fluency. The plurality of factors include at least two of the following: Video shooting conditions, video shooting subjects, video shooting backgrounds, video shooting devices, video resolutions, and video shooting scenes.

[0098] According to an example embodiment of the present disclosure, the training device further comprises: The sampling module is configured to sample the continuous segments of the video sample to obtain continuous video clips sampled from the video sample. The encoding module 702 is configured to: Input the video frames contained in the continuous video clips into a visual encoder to obtain a plurality of spatio-temporal visual features.

[0099] Figure 8 is a block diagram illustrating an evaluation device 800 of video fluency according to an example embodiment of the present disclosure.

[0100] Referring to Figure 8 The video fluency evaluation device 800 can include a target video acquisition module 801, a target feature acquisition module 802, a target compressed feature acquisition module 803, and a fluency evaluation module 804.

[0101] The target video acquisition module 801 is configured to acquire a target video to be evaluated. The target feature acquisition module 802 is configured to input target video frames contained in the target video to be evaluated into a visual encoder to obtain a plurality of target spatio-temporal visual features. The target compressed feature acquisition module 803 is configured to input the plurality of target spatio-temporal visual features into a spatio-temporal visual feature compressor trained according to the training method of the present disclosure to compress the target spatio-temporal visual features, and obtain target compressed spatio-temporal visual features of a preset length. The fluency evaluation module 804 is configured to input the target compressed spatio-temporal visual features of the preset length into an evaluation model trained according to the training method of the present disclosure, to predict the fluency of the target video to be evaluated by the trained evaluation model.

[0102] Figure 9 is a block diagram illustrating an electronic device 900 according to an example embodiment of the present disclosure.

[0103] Referring to Figure 9 The electronic device 900 includes at least one memory 901 having instructions stored therein and at least one processor 902 that, when executing the instructions, performs a method of training an evaluation model for video smoothness according to an example embodiment of the disclosure or a method of evaluating video smoothness.

[0104] As an example, the electronic device 900 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above instructions. Here, the electronic device 900 is not necessarily a single electronic device, but can be a collection of any devices or circuits that can individually or jointly execute the above instructions (or instruction sets). The electronic device 900 can also be a part of an integrated control system or a system manager, or can be configured as a portable electronic device that interfaces with a local or remote (e.g., via wireless transmission).

[0105] In the electronic device 900, the processor 902 can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. As an example and not a limitation, the processor can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0106] The processor 902 can execute instructions or codes stored in the memory 901, and the memory 901 can also store data. The instructions and data can also be transmitted and received via a network through a network interface device, which can use any known transmission protocol.

[0107] The memory 901 can be integrated with the processor 902, for example, by arranging RAM or flash memory within an integrated circuit microprocessor, etc. In addition, the memory 901 can include a separate device, such as an external disk drive, a storage array, or other storage devices that can be used by any database system. The memory 901 and the processor 902 can be operatively coupled or can communicate with each other, for example, through an I / O port, a network connection, etc., so that the processor 902 can read files stored in the memory.

[0108] In addition, the electronic device 900 can also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.). All components of the electronic device 900 can be connected to each other via a bus and / or a network.

[0109] According to an exemplary embodiment of the disclosure, there can also be provided a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the training method of the evaluation model for video smoothness or the evaluation method for video smoothness described above. Examples of the computer-readable storage medium here include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk memory, a hard disk drive (HDD), a solid state drive (SSD), a card-type memory such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card, a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the computer-readable storage medium described above can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc., and, in addition, in one example, the computer program and any associated data, data files, and data structures are distributed over a networked computer system so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0110] According to an exemplary embodiment of the disclosure, there can also be provided a computer program product including a computer program that, when executed by a processor, implements the training method of the evaluation model for video smoothness or the evaluation method for video smoothness according to the disclosure.

[0111] According to the training method, the evaluation method, the device, the electronic device, the storage medium and the computer program product of the video fluency evaluation model, the space-time visual feature compressor is set as a pre-module before the evaluation model, so that the space-time visual feature compressor can be used to change the indefinite length video token input into a fixed length video token input. In this way, the multiple complete videos of indefinite length, i.e., the multiple complete videos of non-fixed length, can be directly used to train the evaluation model, and the frame extraction processing of the videos of various lengths is not required. Since the frame extraction operation is not required before the model training, the video timing information can be avoided to be lost, i.e., the fluency information of the video can be avoided to be lost, and the continuity of the video can be protected. In this way, the model can accurately learn the fluency characteristics between the video frames, and the training effect of the model can be effectively improved, and the accuracy of the fluency evaluation of the trained model can be ensured.

[0112] According to the example embodiments of the present disclosure, the evaluation model can be trained by using the video samples covering multiple factors affecting the video fluency, so that the trained evaluation model can cope with various types of videos, and the generalization ability and robustness of the evaluation model can be ensured.

[0113] According to the example embodiments of the present disclosure, the original video samples can be frame-dropped at multiple preset frame drop rates, and the greater the frame drop rate, the worse the fluency of the corresponding frame-dropped video sample. Therefore, the self-generation of the same-source videos with different degrees of freezing is equivalent to automatically labeling the corresponding fluency of the different frame-dropped video samples. Compared with the manual labeling method, the automatic labeling method provided by the present disclosure can effectively save the labor cost and time cost, and thus the efficiency of labeling can be significantly improved.

[0114] According to the example embodiments of the present disclosure, a degradation method for generating freezing videos and a sorting-based training strategy are provided, i.e., the present disclosure provides a training strategy based on the fluency sorting of the same-source videos, which can greatly expand the fluency training data, and thus the prediction accuracy of the evaluation model can be improved.

[0115] According to the example embodiments of the present disclosure, a preset fluency score benchmark is provided, which can accurately represent the subjective feeling of the user on the video fluency, and thus the fluency video friendly to the user can be accurately recommended to the user, and the viewing experience of the user can be improved.

[0116] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the disclosure be construed as including any paterns of this disclosure that can be derived from the description and illustrations presented herein without departing from the scope and spirit of the disclosure. The specification and examples are exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0117] It is to be understood that the disclosure is not limited to the precise construction described above and shown in the attached drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the disclosure is limited only by the claims that follow.

Claims

1. A method for training a video smoothness evaluation model, characterized in that, The method comprises: obtaining a video sample, wherein the video sample has a smoothness label; inputting a video frame of the video sample into a visual encoder to obtain a plurality of spatio-temporal visual features; inputting the plurality of spatio-temporal visual features into a spatio-temporal visual feature compressor to compress the plurality of spatio-temporal visual features and obtain compressed spatio-temporal visual features of a preset length; inputting the compressed spatio-temporal visual features of the preset length into the evaluation model to obtain a smoothness of the video sample predicted by the evaluation model; calculating a loss based on the predicted smoothness of the video sample and the smoothness label of the video sample; adjusting parameters of the spatio-temporal visual feature compressor and the evaluation model based on the loss.

2. The training method of claim 1, wherein, The method comprises: obtaining an original video sample and a plurality of frame loss video samples corresponding to the original video sample, wherein the original video sample has an absolute smoothness label, and the absolute smoothness label is used to indicate a real smoothness of the original video sample; wherein the step of obtaining a plurality of frame loss video samples comprises: performing frame loss processing on the original video sample according to each of a plurality of preset frame loss rates to obtain the plurality of frame loss video samples, wherein any two video samples in the plurality of frame loss video samples and the original video sample have a relative smoothness label, and the relative smoothness label is used to indicate a high-low contrast relationship between real smoothnesses corresponding to the any two video samples, respectively; the loss is calculated based on the predicted smoothness of the video sample and the smoothness label corresponding to the video sample, comprising: calculating the loss based on the predicted smoothness of the original video sample, the absolute smoothness label, the relative smoothness labels corresponding to the any two video samples, and the predicted smoothness of each video sample in the any two video samples.

3. The training method of claim 2, wherein, the loss is calculated based on the predicted smoothness of the original video sample, the absolute smoothness label, the relative smoothness labels corresponding to the any two video samples, and the predicted smoothness of each video sample in the any two video samples, comprising: calculating a first loss based on the predicted smoothness of the original video sample and the absolute smoothness label; calculating a second loss based on the relative smoothness labels corresponding to the any two video samples and the predicted smoothness of each video sample in the any two video samples; the parameters of the spatio-temporal visual feature compressor and the evaluation model are adjusted based on the loss, comprising: adjusting the parameters of the spatio-temporal visual feature compressor and the evaluation model based on the first loss and the second loss.

4. The training method of claim 2, wherein, the original video sample is divided into T time intervals; ​ ​ According to the number of lost frames corresponding to the preset frame loss rate, at least one video frame is randomly selected from a plurality of video frames in each of the T time intervals and discarded to obtain a lost frame video sample corresponding to the preset frame loss rate.

5. The training method of claim 3, wherein, The first loss is a regression loss, and the second loss is a ranking loss.

6. The training method of claim 1, wherein, The spatio-temporal visual feature compressor comprises an attention network and a feedforward neural network, and the attention network corresponds to a learnable feature representation matrix; The inputting of the plurality of spatio-temporal visual features into the spatio-temporal visual feature compressor to obtain compressed spatio-temporal visual features of a preset length comprises: The plurality of spatio-temporal visual features are inputted into the attention network to obtain an attention result; The attention result is superimposed with the learnable feature representation matrix to obtain a first superimposed result; The first superimposed result is inputted into the feedforward neural network to obtain a feedforward output result; The feedforward output result is superimposed with the first superimposed result to obtain the compressed spatio-temporal visual features of the preset length.

7. The training method of claim 2, wherein, The absolute fluency label corresponding to the original video sample is an absolute fluency label obtained by a user performing fluency scoring on the original video sample based on a preset fluency scoring benchmark, wherein the preset fluency scoring benchmark comprises a plurality of preset scoring levels, and the higher the preset scoring level is, the better the video fluency indicated by the preset scoring level is.

8. The training method of claim 1, wherein, The video samples used for training the evaluation model cover a plurality of factors affecting video fluency; The plurality of factors comprise at least two of the following: Video shooting conditions, video shooting subjects, video shooting backgrounds, video shooting devices, video resolutions, and video shooting scenes.

9. The training method of claim 1, wherein, Before the video frames of the video sample are inputted into a visual encoder to obtain a plurality of spatio-temporal visual features, the training method further comprises: continuous segment sampling of the video sample to obtain a continuous video segment sampled from the video sample; The inputting of the video frames of the video sample into the visual encoder to obtain the plurality of spatio-temporal visual features comprises: inputting the video frames contained in the continuous video segment into the visual encoder to obtain the plurality of spatio-temporal visual features.

10. A method of evaluating video smoothness, characterized by, comprises: acquiring a target video to be evaluated; inputting target video frames contained in the target video to be evaluated into a visual encoder to obtain a plurality of target spatio-temporal visual features; inputting the plurality of target spatio-temporal visual features into a spatio-temporal visual feature compressor trained by the training method in any one of claims 1 to 9 to compress the plurality of target spatio-temporal visual features to obtain target compressed spatio-temporal visual features of a preset length; inputting the target compressed spatio-temporal visual features of the preset length into the evaluation model trained by the training method in any one of claims 1 to 9 to predict the fluency of the target video to be evaluated by the trained evaluation model.

11. A training device for a video smoothness evaluation model, characterized in that, comprises: a sample acquisition module configured to acquire a video sample, wherein the video sample has a fluency label; an encoding module configured to input video frames of the video sample into a visual encoder to obtain a plurality of spatio-temporal visual features; The compression module is configured to input the plurality of spatio-temporal visual features into a spatio-temporal visual feature compressor to compress the plurality of spatio-temporal visual features and obtain compressed spatio-temporal visual features of a preset length; The fluency prediction module is configured to input the compressed spatio-temporal visual features of the preset length into the evaluation model to obtain the fluency of the video sample predicted by the evaluation model; The loss calculation module is configured to calculate a loss based on the predicted fluency of the video sample and a fluency label of the video sample; The parameter adjustment module is configured to adjust parameters of the spatio-temporal visual feature compressor and the evaluation model based on the loss.

12. An apparatus for evaluating video smoothness, the apparatus comprising: a video smoothness evaluator configured to evaluate video smoothness of a video stream based on a video quality metric and a frame rate metric. The target video acquisition module is configured to acquire a target video to be evaluated; The target feature acquisition module is configured to input target video frames contained in the target video to be evaluated into a visual encoder to obtain a plurality of target spatio-temporal visual features; The target compressed feature acquisition module is configured to input the plurality of target spatio-temporal visual features into the spatio-temporal visual feature compressor trained by the training method of any one of claims 1 to 9 to compress the plurality of target spatio-temporal visual features and obtain target compressed spatio-temporal visual features of a preset length; The fluency evaluation module is configured to input the target compressed spatio-temporal visual features of the preset length into the evaluation model trained by the training method of any one of claims 1 to 9 to predict the fluency of the target video to be evaluated by the trained evaluation model. The processor; 13. An electronic device, comprising: The memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the evaluation model of the video fluency according to any one of claims 1 to 9, or implement the evaluation method of the video fluency according to claim 10. When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute the training method of the evaluation model of the video fluency according to any one of claims 1 to 9, or execute the evaluation method of the video fluency according to claim 10. The computer program is executed by the processor to implement the training method of the evaluation model of the video fluency according to any one of claims 1 to 9, or implement the evaluation method of the video fluency according to claim 10.

14. A computer-readable storage medium, characterized in that, ​ 15. A computer program product comprising a computer program, characterized in that, ​

Images