Method and device for evaluating video quality and method and device for training video quality evaluation model

Abstract

The embodiment of the invention provides a method and device for evaluating video quality and training a video quality evaluation model, and the method comprises the steps: obtaining a first segment set and a second segment set, the first segment set is obtained through sampling a first video segment, and the second segment set is obtained through sampling a second video segment; the second clip set is obtained by sampling the downsampled first video clip obtained by a weighted average-based sampling mode; determining a key frame sequence based on the first video; based on the similarity between each frame block in any first key frame and the first key frame, screening from the patch blocks of the corresponding first fragment and second fragment to obtain a corresponding third fragment, the patch blocks of the first segment and the second segment corresponding to the first key frame respectively have a corresponding relation with each frame block in the first key frame; determining a first video feature based on the third segment corresponding to each key frame; and obtaining a quality evaluation result corresponding to the first video based on the first video feature, thereby realizing accurate evaluation of the video quality.

Description

Technical Field

[0001] This specification relates to the technical field of video processing, and in particular, to a method and apparatus for evaluating video quality and training a video quality evaluation model. Background Art

[0002] As a new media form, short videos have become rapidly popular on various social media platforms such as short video platforms in recent years. Due to the characteristics of short videos, such as short duration, diverse content, and fast rhythm, the number of users and the amount of content, i.e., the number of short videos, on short video platforms has increased explosively. Among them, the short video content on short video platforms is generally user-generated content (UGC, User Generated Content), that is, video content recorded by non-professional enthusiasts using devices with various performances. Since the devices used by short video users to record video content have different performances, and the number of short video content on short video platforms is large, there is a need for automatic evaluation of short video quality. Therefore, how to evaluate its quality has become an important research topic and application requirement. Summary of the Invention

[0003] One or more embodiments of this specification provide a method and apparatus for evaluating video quality and training a video quality evaluation model to achieve accurate evaluation of video quality.

[0004] According to a first aspect, a method for evaluating video quality is provided. The method includes:

[0005] Obtain a first fragment set and a second fragment set, where the first fragment set is obtained by fragment sampling of a first video, and the second fragment set is obtained by performing the fragment sampling on the downsampled first video obtained based on a specified sampling method, and the specified sampling method is a sampling method based on weighted average;

[0006] Based on the first video, determine a key frame sequence, where each key frame in the key frame sequence has a corresponding relationship with each first fragment and each second fragment;

[0007] For any first key frame among each key frame, based on the similarity between each frame block in the first key frame and the first key frame, screen out a third fragment corresponding to the first key frame from the patch blocks of its corresponding first fragment and second fragment, and the patch blocks of the corresponding first fragment and second fragment of the first key frame respectively have a corresponding relationship with each frame block in the first key frame;

[0008] Based on the third fragments corresponding to each key frame, determine a first video feature;

[0009] Based on the first video feature, obtain a quality evaluation result corresponding to the first video.

[0010] According to a second aspect, a method for training a video quality evaluation model is provided. The video quality evaluation model includes: a backbone network and a quality prediction network, including:

[0011] Obtain the quality label result, the first sample segment set, and the second sample segment set corresponding to the sample video. Among them, the first sample segment set is obtained by segment sampling the sample video, and the second sample segment set is obtained by segment sampling the downsampled sample video obtained based on a specified sampling method. The specified sampling method is a sampling method based on weighted average;

[0012] Based on the sample video, determine the sample key frame sequence. Among them, each sample key frame in the sample key frame sequence has a corresponding relationship with each first sample segment and each second sample segment;

[0013] For any first sample key frame in each sample key frame, based on the similarity between each frame block in the first sample key frame and the first sample key frame, filter out the third sample segment corresponding to the first sample key frame from the patch blocks of its corresponding first sample segment and second sample segment respectively. The patch blocks of the corresponding first sample segment and second sample segment of the first sample key frame have a corresponding relationship with each frame block in the first sample key frame;

[0014] Based on the third sample segment corresponding to each sample key frame, determine the target video feature through the backbone network;

[0015] Based on the target video feature, obtain the quality prediction result corresponding to the sample video through the quality prediction network;

[0016] Train the backbone network based on the difference between the quality prediction result and the quality label result.

[0017] According to a third aspect, a device for evaluating video quality is provided. The device includes:

[0018] A first acquisition module configured to acquire a first segment set and a second segment set. Among them, the first segment set is obtained by segment sampling the first video, and the second segment set is obtained by segment sampling the downsampled first video obtained based on a specified sampling method. The specified sampling method is a sampling method based on weighted average;

[0019] A first determination module configured to determine a key frame sequence based on the first video. Among them, each key frame in the key frame sequence has a corresponding relationship with each first segment and each second segment;

[0020] The first screening module is configured to, for any first key frame among each key frame, based on the similarity between each frame block in the first key frame and the first key frame, screen and obtain a third segment corresponding to the first key frame from the patch blocks of its corresponding first segment and second segment respectively, and the patch blocks of the first segment and second segment corresponding to the first key frame respectively have a corresponding relationship with each frame block in the first key frame;

[0021] The second determination module is configured to determine a first video feature based on the third segment corresponding to each key frame;

[0022] The first obtaining module is configured to obtain a quality evaluation result corresponding to the first video based on the first video feature.

[0023] According to the fourth aspect, there is provided an apparatus for training a video quality evaluation model, where the video quality evaluation model includes: a backbone network and a quality prediction network, including:

[0024] The second obtaining module is configured to obtain a quality label result corresponding to a sample video, a first sample segment set, and a second sample segment set, where the first sample segment set is obtained by performing segment sampling on the sample video, and the second sample segment set is obtained by performing the segment sampling on the downsampled sample video obtained based on a specified sampling method, and the specified sampling method is a sampling method based on weighted average;

[0025] The third determination module is configured to determine a sample key frame sequence based on the sample video, where each sample key frame in the sample key frame sequence has a corresponding relationship with each first sample segment and each second sample segment respectively;

[0026] The second screening module is configured to, for any first sample key frame among each sample key frame, based on the similarity between each frame block in the first sample key frame and the first sample key frame, screen and obtain a third sample segment corresponding to the first sample key frame from the patch blocks of its corresponding first sample segment and second sample segment respectively, and the patch blocks of the first sample segment and second sample segment corresponding to the first sample key frame respectively have a corresponding relationship with each frame block in the first sample key frame;

[0027] The fourth determination module is configured to determine a target video feature through the backbone network based on the third sample segment corresponding to each sample key frame;

[0028] The second obtaining module is configured to obtain a quality prediction result corresponding to the sample video through the quality prediction network based on the target video feature;

[0029] A training module, configured to train the backbone network based on the difference between the quality prediction result and the quality label result.

[0030] According to a fifth aspect, there is provided a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute the method described in the first aspect or the second aspect.

[0031] According to a sixth aspect, there is provided a computing device, including a memory and a processor. Among them, an executable code is stored in the memory. When the processor executes the executable code, the method described in the first aspect or the second aspect is implemented.

[0032] According to the method and device provided in the embodiments of this specification, a first fragment set obtained by performing segment sampling on a first video and a second fragment set obtained by performing segment sampling on a downsampled first video obtained based on a specified sampling method are acquired. Since the specified sampling method is a sampling method based on weighted average, correspondingly, each patch block in each second fragment in the second fragment set aggregates more information in the corresponding video frame block. In this way, based on the similarity between each frame block in the first key frame and the first key frame respectively, a third fragment corresponding to the first key frame is screened from the patch blocks of its corresponding first fragment and second fragment respectively. Among the third fragments corresponding to each key frame obtained in this way, the continuity and integrity in space of the patch blocks included therein with respect to the corresponding video frame are not damaged, and the patch blocks screened from the second fragment in the third fragment can contain more information in the original video frame block, providing more useful information for the subsequent evaluation of the video quality. Using such a third fragment, a quality evaluation result corresponding to the first video with better accuracy can be obtained to achieve a relatively accurate evaluation of the quality of the first video. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0034] Figure 1 It is a schematic diagram of the implementation framework of an embodiment disclosed in this specification;

[0035] Figure 2 It is a schematic flowchart of a method for training a video quality evaluation model provided in the embodiment;

[0036] Figure 3A It is a schematic diagram of a scenario of segment sampling provided in the embodiment;

[0037] Figure 3B Another schematic diagram of fragment sampling provided for the embodiment;

[0038] Figure 3C A schematic diagram of a scenario for determining the third sample fragment provided for the embodiment;

[0039] Figure 4 A flowchart of a method for evaluating video quality provided for the embodiment;

[0040] Figure 5 An exemplary structural diagram of the second branch network provided for the embodiment;

[0041] Figure 6 An exemplary structural diagram of the backbone network provided for the embodiment;

[0042] Figure 7 A flowchart of a method for evaluating video quality provided for the embodiment;

[0043] Figure 8 A flowchart of a method for evaluating video quality provided for the embodiment;

[0044] Figure 9 A schematic block diagram of a device for evaluating video quality provided for the embodiment;

[0045] Figure 10 A schematic block diagram of a device for training a video quality evaluation model provided for the embodiment. Detailed implementation manners

[0046] Next, the technical solutions of the embodiments of the present specification will be described in detail with reference to the accompanying drawings.

[0047] The embodiments of the present specification disclose a method and a device for evaluating video quality and training a video quality evaluation model. First, the application scenarios and technical concepts of the method will be introduced as follows:

[0048] As mentioned above, short videos, as a new media form, have become rapidly popular on various social media platforms such as short video platforms in recent years. The short video content on short video platforms is generally user-generated content (UGC, User Generated Content), that is, video content recorded by non-professional enthusiasts using devices of various performances. Due to the different performances of the devices used by short video users to record video content and the large number of short video contents on short video platforms, there is a need for automatic evaluation of short video quality. Then, how to evaluate its quality has become an important research topic and application requirement.

[0049] In view of this, the inventors propose a method for evaluating video quality and training a video quality evaluation model. Figure 1 Fig. Figure 1 shows a schematic diagram of an implementation scenario according to an embodiment disclosed in this specification. In this implementation scenario, a trained video quality evaluation model is used to evaluate the quality of a video. The video quality evaluation model at least includes a backbone network and a quality prediction network. The backbone network is used to extract image features, and the quality prediction network is used to predict the video quality.

[0050] During the quality evaluation process, a first video to be evaluated for quality is obtained. The first video can be any type of video, such as a short video or a clip of a movie or TV drama. The first video includes multiple video frames, which can also be referred to as a video frame sequence. A first segment set and a second segment set corresponding to the first video are obtained. The first segment set is obtained by segment sampling the first video, and the second segment set is obtained by segment sampling the downsampled first video obtained by a specified sampling method. The specified sampling method is a sampling method based on weighted average. It can be understood that the first segment set includes multiple first segments, and each first segment corresponds to a video frame in the first video. The second segment set includes multiple second segments, and each second segment corresponds to a video frame in the downsampled first video. There is a corresponding relationship between each first segment in the first segment set and each second segment in the second segment set. Exemplarily, the first segment a corresponds to the video frame b in the first video, and the second segment c corresponding to the first segment a corresponds to the video frame d in the downsampled first video, where the video frame d is the downsampled video frame b.

[0051] Exemplarily, the foregoing segment sampling can be a segment sampling method based on the GMS (Grid-based Motion Statistics) algorithm. The foregoing specified sampling method is a sampling method based on weighted average. Such a sampling method can enable each pixel in the downsampled first video to aggregate the image information of the corresponding region in the first video of the original size.

[0052] Next, based on the first video, a key frame sequence is determined. Each key frame in the key frame sequence has a corresponding relationship with each first segment in the first segment set and also has a corresponding relationship with each second segment in the second segment set. Then, for any first key frame in the key frame sequence, the similarity between each frame block in the first key frame and the first key frame is calculated. To some extent, this similarity can reflect the importance and contribution degree of each frame block to determining the quality result of the first video. The patch blocks of the first segment and the second segment corresponding to the first key frame respectively have a corresponding relationship with each frame block in the first key frame. Correspondingly, the importance and contribution degree of each frame block to determining the quality result of the first video can, to some extent, characterize the importance and contribution degree of the patch blocks of the first segment and the second segment corresponding to the first key frame to determining the quality result of the first video.

[0053] In some possible examples, such as Figure 1 shown, the video quality evaluation model further includes a first branch network. The similarity between each frame block in the aforementioned first key frame and the first key frame can be determined based on the first branch network. Exemplarily, based on the first key frame, through the first branch network, the global visual feature corresponding to the first key frame and the local visual features corresponding to each frame block therein can be determined. Then, based on the global visual feature corresponding to the first key frame and the local visual features corresponding to each frame block therein, the similarity between the local visual features corresponding to each frame block in the first key frame and its global visual feature is determined as the similarity between each frame block in the first key frame and the first key frame.

[0054] Then, based on the similarity between each frame block in the first key frame and the first key frame, appropriate patch blocks are selected from the patch blocks of its corresponding first segment and second segment respectively to combine and obtain the third segment corresponding to the first key frame. Exemplarily, in order to make the third segments corresponding to each key frame obtained retain more information beneficial to video quality evaluation, M patch blocks corresponding to the M frame blocks with the largest corresponding similarity can be determined from the second segment corresponding to the first key frame. And the patch blocks corresponding to the other frame blocks (i.e., the other frame blocks in the first key frame except the aforementioned M frame blocks) are determined from the first segment corresponding to the first key frame to combine and obtain the third segment corresponding to the first key frame. The spatial continuity between the patch blocks in the third segment obtained in this way is better, and among the M patch blocks corresponding to the M frame blocks with the largest corresponding similarity, more information beneficial to the video quality evaluation result is included.

[0055] Then, based on the third segments corresponding to each key frame, the first video feature is determined through the backbone network. And based on the first video feature, the quality evaluation result corresponding to the first video is obtained through the quality prediction network.

[0056] In some possible examples, to better improve the accuracy of the quality evaluation result. For example Figure 1 As shown, the video quality evaluation model may further include a second branch network. Correspondingly, based on the first video, the time features corresponding to the first video are determined through the second branch network; then, in the process of determining the first video features, by combining each third segment and the aforementioned time features, the first video features are obtained through the backbone network. In this way, the first video features obtained increase the information in the time dimension of the first video obtained through the second branch network, providing a basis for obtaining a quality evaluation result with better accuracy; furthermore, based on such first video features, through the quality prediction network, a quality evaluation result corresponding to the first video with relatively higher accuracy is obtained.

[0057] In the above process, based on the similarity between each frame block in the first key frame and the first key frame, the third segment corresponding to the first key frame is screened from the patch blocks of its corresponding first segment and second segment respectively. Among the third segments corresponding to each key frame obtained in this way, the patch blocks included therein have not been damaged in terms of spatial continuity and integrity with respect to the corresponding video frames, and the patch blocks screened from the second segment in the third segment can contain more information in the original video frame blocks, providing more useful information for the subsequent evaluation of the video quality. Using such third segments through the backbone network and the quality prediction network, a quality evaluation result corresponding to the first video with better accuracy can be obtained, so as to realize a relatively accurate evaluation of the quality of the first video.

[0058] Next, in combination with specific embodiments, the methods and apparatuses for evaluating video quality and training a video quality evaluation model provided in this specification will be elaborated in detail.

[0059] First, the process of training a video quality evaluation model will be introduced below.

[0060] Figure 2 The flowchart of the method for training a video quality evaluation model in an embodiment of this specification is shown. This method is executed by an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. In some examples, the video quality evaluation model at least includes: a backbone network and a quality prediction network. In the process of training the video quality evaluation model, as Figure 2 shown, the method includes the following steps S210 - S260:

[0061] In step S210, obtain the quality label result corresponding to the sample video, the first sample segment set, and the second sample segment set. The first sample segment set is obtained by segment sampling the sample video, and the second sample segment set is obtained by segment sampling the downsampled sample video obtained based on a specified sampling method, where the specified sampling method is a sampling method based on weighted average.

[0062] In some possible implementation manners, the electronic device obtains a training set for training a video quality evaluation model. The training set includes a number of sample data. Any sample data X may include the quality label result corresponding to the sample video X, the first sample segment set, and the second sample segment set. The electronic device can obtain any sample data from the training set to perform the subsequent process of training the video quality evaluation model. It can be understood that the process of training the video quality evaluation model using each sample data is similar. Below, taking any one of the sample data X as an example, the process of training the video quality evaluation model is described. For the process of training the video quality evaluation model using other sample data, reference can be made to the process of training the video quality evaluation model using the sample data X, which will not be elaborated here.

[0063] It can be understood that the first sample segment set corresponding to the sample video X in the sample data X is obtained by segment sampling the sample video X, and the second sample segment set is obtained by segment sampling the downsampled sample video X' obtained based on a specified sampling method, where the specified sampling method is a sampling method based on weighted average. In some examples, the specified sampling method may include, but is not limited to, sampling methods based on weighted average such as bilinear interpolation algorithm or trilinear interpolation algorithm. By downsampling the video frames in such a sampling method, the pixel values of each pixel in the downsampled video frames can aggregate more information of the original-sized video.

[0064] Assume that it is necessary to downsample each video frame with a size of N*N in the sample video X to a video frame of (N / 4)*(N / 4) based on the specified sampling method to obtain the downsampled sample video X'. Correspondingly, each pixel in each video frame in the sample video X' can aggregate the information of the pixels in the corresponding 4*4 regions of each video frame in the sample video X.

[0065] In the embodiments of this specification, when downsampling the sample video X, the aspect ratio of the downsampled sample video X' can be kept the same as the original aspect ratio of the sample video X.

[0066] The following describes a process of performing segment sampling on a sample video to obtain a first sample segment set. Exemplarily, the process may include: using a sampling method based on the GMS algorithm, performing segment sampling on each sample video frame in the sample video X to obtain a segment corresponding to each sample video frame, which is hereinafter referred to as a first sample segment corresponding to each sample video frame, so that the first sample segment obtained can better maintain the original texture and spatial structure in the sample video frame.

[0067] After obtaining the first sample segments corresponding to each sample video frame, a first sample segment set is obtained based on the first sample segments corresponding to each sample video frame. Here, taking any sample video frame p in the sample video X as an example, the segment sampling process is described.

[0068] The segment sampling process includes, for example: dividing the sample video frame p into G f *G f grids, and then randomly sample S in each grid f *S f The pixel blocks of G are called patches to keep the local texture features. f *G f The G sampled from the grid f *G f The size is S f *S f The patches are arranged into a grid of size (G f *S f )*(G f *S f ), that is, the first sample segment corresponding to the sample video frame p is obtained. Similarly, the first sample segment corresponding to each sample video frame in the sample video is obtained, and then the first sample segment set is formed. Figure 3A The schematic relationship between the sample video frame p and its corresponding first sample segment is shown as an example, wherein G f *G f It is set to 3*3, and the area corresponding to each gray box in each grid of the sample video frame p is the patch block sampled from the grid.

[0069] It should be noted that, in order to ensure the alignment of the first sample segments corresponding to each sample video frame p in the time dimension, for any sample video frame p and sample video frame q in the sample video, the grid division between them is the same, and when randomly sampling patch blocks in any a-th grid in the sample video frame p and the sample video frame q, the relative positions of the sampled patch blocks on their respective sample video frames are the same, such as Figure 3A As shown, where p is not equal to q.

[0070] Exemplarily, in order to reduce the computational amount during the model training process and subsequent inference process, after obtaining the sample video, the sample video can also be frame-sampled to reduce the number of video frames in the video processed by the model. Correspondingly, the sample video is divided into a specified number of uniform and non-overlapping sample video segments, for example, obtaining G1 first sample video segments; for each first sample video segment, video frame sampling is performed on it to obtain multiple video frames corresponding to each first sample video segment. Subsequently, the foregoing segment sampling process is adopted to perform segment sampling on the multiple video frames corresponding to each first sample video segment to obtain a first sample segment set. In this way, the number of video frames input to the model can be reduced, the computational amount during the model training process and subsequent inference process can be reduced, and the speed of model training and the speed of subsequent inference process can be improved.

[0071] In some specific examples, in order to retain the original changes in the time dimension of the video, when performing video frame sampling for each sample video segment, continuous frame sampling can be performed, that is, the multiple video frames sampled for each sample video segment are continuous.

[0072] In the process of performing segment sampling on the downsampled sample video to obtain a second sample segment set, reference can be made to the process of performing segment sampling on the sample video to obtain a first sample segment set, where only the sample video is replaced by the downsampled sample video. It can be understood that there is a corresponding relationship between each second sample segment in the second sample segment set and each first sample segment in the first sample segment set. Among them, for the second sample segment x′ in the second sample segment set that has a corresponding relationship with the first sample segment x, the corresponding sample video frame is the video frame after downsampling of the sample video frame corresponding to the first sample segment x.

[0073] The sample video frame p corresponding to the first sample segment x is as Figure 3B shown, and the sample video frame p′ corresponding to the second sample segment x′ that has a corresponding relationship with the first sample segment x is as Figure 3B shown. Among them, there is a one-to-one corresponding relationship between each patch block in the first sample segment x and each patch block in the second sample segment x′. The patch block in the first sample segment x and the patch block in the second sample segment x′ that have a corresponding relationship are related in their positions in their respective grids. For example, their relative positions in their respective grids are the same.

[0074] It can be understood that the sizes of the second sample segments can be the same as the sizes of the first sample segments. For example, each second sample segment respectively includes G f *G f patch blocks with a size of S f *S f .

[0075] In step S220, based on the sample video, a sample key frame sequence is determined, where each sample key frame in the sample key frame sequence has a corresponding relationship with each first sample segment and each second sample segment respectively.

[0076] In some possible examples, the sample video can be divided to obtain a plurality of second sample video segments, where the second sample video segments may not overlap with each other. The number of the obtained second sample video segments may be related to the number of segments in the foregoing first sample segment set, that is, related to the number of segments in the foregoing second sample segment set.

[0077] Subsequently, in one case, for any second sample video segment di in the plurality of second sample video segments, the average value between the first video frame and the last video frame therein can be used as the sample key frame corresponding to the second sample video segment di, and so on, to obtain the sample key frames corresponding to the second sample video segments, thereby obtaining the sample key frame sequence.

[0078] In another case, for any second sample video segment di in the plurality of second sample video segments, the first video frame or the last video frame therein can be used as the sample key frame corresponding to the second sample video segment di, and so on, to obtain the sample key frames corresponding to the second sample video segments, thereby obtaining the sample key frame sequence. Wherein, each sample key frame in the sample key frame sequence has a corresponding relationship with each first sample segment and each second sample segment respectively.

[0079] In some other possible examples, the multiple video frames corresponding to the foregoing obtained first sample video segments can also be used as the respective sample key frames to obtain the sample key frame sequence.

[0080] Next, in step S230, for any first sample key frame among the sample key frames, based on the similarity between each frame block in the first sample key frame and the first sample key frame itself, a third sample segment corresponding to the first sample key frame is screened from the patch blocks of its corresponding first sample segment and second sample segment respectively, where the patch blocks of the first sample segment and the second sample segment corresponding to the first sample key frame respectively have a corresponding relationship with each frame block in the first sample key frame.

[0081] In this step, for any first sample key frame among the sample key frames in the sample key frame sequence, the electronic device can calculate the similarity between each frame block in the first sample key frame and the first sample key frame, and then, based on the similarity between each frame block in the first sample key frame and the first sample key frame, screen appropriate patch blocks from the patch blocks of its corresponding first sample segment and second sample segment respectively, so as to obtain a third sample segment corresponding to the first sample key frame. Among them, the similarity between each frame block and the first sample key frame can be reflected by a similarity value. Among them, the larger the similarity value is, the more similar the corresponding frame block is to the first sample key frame.

[0082] In some possible examples, M largest similarities (subsequently referred to as first similarities) can be determined from the similarities between each frame block in the first sample key frame and the first sample key frame, and patch blocks corresponding to the frame blocks corresponding to the M first similarities can be determined from the patch blocks of the second sample segment corresponding to the first sample key frame; and patch blocks corresponding to the frame blocks corresponding to each second similarity can be determined from the patch blocks of the first sample segment corresponding to the first sample key frame, where the second similarity is the other similarities except the M first similarities among all similarities. The value of M can be set according to requirements. For example, it can be set to half of the total number A of frame blocks in the first sample key frame, that is, M = (A / 2). Among them, when A is odd, the value of M can be rounded up or down to (A / 2).

[0083] In the above manner, the patch blocks corresponding to the frame blocks corresponding to the M first similarities and the patch blocks corresponding to the frame blocks corresponding to each second similarity are continuous in space; and each patch block of the second sample segment can contain more information about the local area of the first sample key frame, that is, the information within each grid. It can be understood that the greater the similarity to the first sample key frame, the stronger the correlation of the frame block to the quality of the first sample key frame. Determining the patch blocks corresponding to the frame blocks corresponding to the M largest first similarities from the patch blocks of the second sample segment can help retain more information with stronger correlation to the quality of the first sample key frame, providing a basis for the accuracy of the subsequent video quality evaluation result.

[0084] Such as Figure 3CAs shown, assume that the first sample key frame includes 3*3 frame blocks, and the similarity of each of frame blocks 1-9 to the first sample key frame decreases in sequence; the first sample segment and the second sample segment corresponding to the first sample key frame each include 3*3 patch blocks, and assume that M is equal to 5. Correspondingly, 5 patch blocks can be determined from the patch blocks of the second sample segment, and these 5 patch blocks respectively correspond to the 5 frame blocks with the greatest similarity in the first sample key frame, that is, frame blocks 1-5, as Figure 3B shown, the patch block 11 corresponding to frame block 1, the patch block 12 corresponding to frame block 2, the patch block 13 corresponding to frame block 3, the patch block 14 corresponding to frame block 4, and the patch block 15 corresponding to frame block 5 in the second sample segment; 4 patch blocks can be determined from the patch blocks of the first sample segment, and these 4 patch blocks respectively correspond to the 4 frame blocks with the smallest similarity in the first sample key frame, that is, frame blocks 6-9, as Figure 3C shown, the patch block 26 corresponding to frame block 6, the patch block 27 corresponding to frame block 7, the patch block 28 corresponding to frame block 8, and the patch block 29 corresponding to frame block 9 in the first sample segment.

[0085] Through the aforementioned patch blocks 11, 12, 13, 14, and 15, and patch blocks 26, 27, 28, and 29, a third sample segment is combined, as Figure 3C shown.

[0086] In some possible examples, as Figure 1 shown, the video quality evaluation model may further include a first branch network, which can extract spatial dimension information from video frames for enhancing the features provided by the backbone network in the spatial dimension. In some examples, the first branch network can be implemented at least through the visual encoding network of the multi-modal model CLIP (Contrastive Language-Image Pre-training). The visual encoding network has been pre-trained based on a specified set of text-image pairs.

[0087] Correspondingly, before screening the third sample segment corresponding to the first sample key frame in step S230, steps 11-12 may further be included:

[0088] In step 11, based on the first sample key frame, through the first branch network, determine the global visual feature corresponding to the first sample key frame and the local visual features corresponding to each frame block therein. In this step, input the first sample key frame into the first branch network so that the first branch network processes the first sample key frame to determine the visual feature corresponding to the first sample key frame, and then based on the visual feature corresponding to the first sample key frame, determine the global visual feature corresponding to the first sample key frame and the local visual features corresponding to each frame block therein.

[0089] In some other examples, in order to better maintain robustness, before processing the first sample key frame using the first branch network, the first sample key frame can also be preprocessed to obtain a frame after image preprocessing. Then, based on the frame after image preprocessing, through the first branch network, the global visual features corresponding to the first sample key frame and the local visual features corresponding to each frame block therein are determined. Among them, the image preprocessing can include data augmentation operations such as random rotation and random cropping, and can also include size adjustment operations, such as adjusting the size of the first sample key frame after data augmentation to a specified size, such as adjusting it to 224*224*3.

[0090] In some embodiments, as Figure 1 shown, the first branch network includes a visual encoding network and a feature mapping network;

[0091] In step 11, it may include the following steps 111-113: In step 111, based on the first sample key frame, through the visual encoding network, sample encoding features are obtained. In this step, the first sample key frame or the first sample key frame after the aforementioned image preprocessing and size adjustment is input into the visual encoding network to process its input through the visual encoding network to obtain sample encoding features.

[0092] Then, in step 112, based on the sample encoding features, through the feature mapping network, sample mapping features are obtained. In this step, the sample encoding features are input into the feature mapping network to process the sample encoding features through the feature mapping network to obtain sample mapping features. Exemplarily, the feature mapping network can be implemented as a linear layer or a convolutional layer.

[0093] Next, in step 113, based on the sample mapping features, the global sample features corresponding to the first sample key frame and the local sample features corresponding to each frame block therein are determined.

[0094] In some possible examples, the electronic device can determine the local sample features corresponding to each frame block from the sample mapping features; then, based on the mean value of the local sample features corresponding to each frame block, the global sample features corresponding to the first sample key frame can be determined. It can be understood that the sample mapping features are determined based on the first sample key frame, and there is a corresponding relationship between its local sample features and each frame block in the first sample key frame.

[0095] Then, in step 12, determine the similarity between the local visual features corresponding to each frame block in the first sample key frame and its global visual feature respectively. In this step, a specified similarity algorithm can be used to determine the similarity between the local visual features corresponding to each frame block in the first sample key frame and its global visual feature respectively, where the similarity can characterize the contribution degree of each frame block to the video quality evaluation result. The greater the similarity, the greater the contribution degree of the corresponding frame block to the video quality evaluation result. The specified similarity algorithm can be any algorithm for calculating the similarity between two data in related technologies.

[0096] In some specific examples, the similarity between the local visual features corresponding to each frame block in the first sample key frame and its global visual feature respectively can be determined by the following formula (1):

[0097]

[0098] where S u represents the similarity between the local visual feature corresponding to the u-th frame block in the first sample key frame and the global visual feature of the first sample key frame; f C represents the global visual feature of the first sample key frame, and f pu represents the local visual feature corresponding to the u-th frame block in the first sample key frame. The u is a positive integer, and its maximum value is the number of frame blocks in the first sample key frame, for example, the aforementioned G f *G f .

[0099] After obtaining the similarity between the local visual features corresponding to each frame block in the first sample key frame and the global visual feature of the first sample key frame respectively, in step S230, specifically, determine M patch blocks corresponding to the M local visual features with the greatest corresponding similarity from the second sample segment corresponding to the first sample key frame; and determine each patch block corresponding to the other local visual features from the first sample segment corresponding to the first sample key frame, so as to obtain a third sample segment corresponding to the first sample key frame.

[0100] In this step, each patch of the second sample segment may contain more information about the local regions of the first sample key frames, i.e., the information within each grid. From these, M patches corresponding to the M local visual features with the greatest similarity are determined, and for each of the other local visual features, the corresponding patches are determined from the first sample segment corresponding to the first sample key frame. Then, the two sets of determined patches are combined to obtain the third sample segment corresponding to the first sample key frame. The third sample segment obtained in this way can ensure the continuity between patches and retain more information that contributes to the quality evaluation result, providing a basis for ensuring the accuracy of the subsequent quality evaluation result.

[0101] Through the above method, the third sample segment corresponding to each sample key frame in the sample key frame sequence can be obtained. Then, in step S240, based on the third sample segment corresponding to each sample key frame, the target video feature is determined through the backbone network. In this step, the third sample segments corresponding to each sample key frame can be combined into a third sample segment sequence and input into the backbone network. By processing the third sample segment sequence through the backbone network, the target video feature is obtained.

[0102] Among them, the backbone network is used to extract the features of the video frame sequence, and it can be any network in the related art that can process the video frame sequence. In some embodiments, the backbone network can be a Fragment Attention Network (FANet). The network architecture of the backbone network can be the architecture of 3D Swin Transformer. In this architecture, for a segment, considering that some pixel pairs in the same attention window may be within the same patch or in different patches, where the pixel pair can refer to the pixel pair for calculating the attention. To better represent the relative positions between different pixels, FANet uses two learnable real position deviation tables and pseudo position deviation tables to replace the original gating position relative deviation of the deviation, so as to better extract the features of the segment and obtain a more appropriate target video feature.

[0103] Then, in step S250, based on the target video feature, the quality prediction result corresponding to the sample video is obtained through the quality prediction network.

[0104] In some possible examples, the electronic device can input the target video feature into the quality prediction network to process the target video feature through the quality prediction network and obtain the quality prediction result corresponding to the sample video. Exemplarily, the quality prediction result may include a quality score, where the larger the quality score, the higher the quality of the sample video predicted by the video quality evaluation model. Exemplarily, the quality prediction network can be implemented as a fully connected layer such as a multi-layer perceptron or a network based on the Transformer structure.

[0105] After obtaining the quality prediction result corresponding to the sample video, in step S260, the backbone network is trained based on the difference between the quality prediction result and the quality label result.

[0106] In this step, a preset loss function can be used to determine the model loss value based on the difference between the quality prediction result and the quality label result. Then, with the goal of minimizing the model loss value, the parameters of the backbone network are adjusted to train the backbone network. Exemplarily, the electronic device can determine the model loss value by using the preset loss function based on the difference between the quality prediction result and the quality label result corresponding to each obtained sample video. Another example is that the electronic device can also determine the model loss value by using the preset loss function based on the differences between the quality prediction results and the quality label results corresponding to multiple sample videos after obtaining them.

[0107] In some possible examples, the preset loss function can be any type of loss function, such as but not limited to cross-entropy loss function, mean square error loss function, and loss function based on PLCC (Pearson’s linear correlation coefficient). Among them, the loss function based on PLCC can be expressed as PLCC loss.

[0108] PLCC can be used to maximize the linear correlation between the quality score predicted by the model and the true quality score, such as maximizing the linear correlation between the aforementioned quality prediction result and the quality label result. Among them, the closer the PLCC value is to 1, the stronger the linear relationship between the quality score predicted by the model and the true quality score, and the better the corresponding model performance. When the preset loss function is PLCC loss, the electronic device can determine the quality prediction results and quality label results corresponding to multiple sample videos through the aforementioned steps S210 - S250. Then, use PLCC loss to jointly determine the model loss value based on the differences between the quality prediction results and the quality label results corresponding to multiple sample videos.

[0109] Among them, the model loss value can be expressed by the following formula (3):

[0110]

[0111] Where represents the quality score predicted by the model, that is, the quality prediction result corresponding to the i-th sample video among multiple sample videos, y iis the true quality score, i.e., the quality label result corresponding to the i-th sample video among multiple sample videos, and respectively represent the mean of multiple quality label results y corresponding to multiple sample videos, and the mean of multiple quality prediction results corresponding to multiple sample videos, where N represents the number of multiple sample videos.

[0112] In some of the examples described above, the video quality evaluation model may further include the aforementioned first branch network, where the first branch network includes a video coding network and a feature mapping network; in the case of such a model structure, in order to better improve the accuracy of the model results, in step S260, the following steps 21-23 may be included:

[0113] In step 21, based on the sample key frame sequence, through a specified video scoring network, the key frame features corresponding to each sample key frame are obtained. In some embodiments, the specified video scoring network may be a multi-modal network that is pre-trained based on image-text pairs and video-text pairs. The text in the image-text pairs and video-text pairs is generally used to describe the quality of the corresponding image or video. In some specific examples, the specified video scoring network may be implemented by the Q-Align model.

[0114] In some examples, the electronic device may input the sample key frame sequence into the specified video scoring network to process the sample key frame sequence through the specified video scoring network to obtain the extracted features corresponding to each sample key frame, hereinafter referred to as key frame features. In some other examples, the electronic device may also first preprocess each sample key frame in the sample key frame sequence to obtain a preprocessed sample key frame sequence, and then input the preprocessed sample key frame sequence into the specified video scoring network to obtain the key frame features corresponding to each sample key frame. Among them, the preprocessing may include, but is not limited to, various image enhancement operations and size adjustment operations.

[0115] Next, in step 22, using the KL divergence algorithm, based on the global sample features and key frame features corresponding to each sample key frame, the KL divergence value is determined.

[0116] Considering that the specified video scoring network is trained and performs well in tasks such as image quality assessment, video quality assessment, and aesthetic expression; and, the specified video scoring network is trained using image-text pairs and video-text pairs, and the features obtained through its encoding can contain text semantic information related to its quality. In this way, the key frame features obtained through its encoding can carry more features related to video quality assessment. In view of this, in order to better improve the accuracy of the quality assessment results, the output of the specified video scoring network can be used to constrain the global sample features in the output of the first branch network, so that the distribution of the global sample features in the output of the first branch network is closer to the distribution of the key frame features corresponding to each sample key frame obtained through the encoding of the specified video scoring network.

[0117] Correspondingly, the KL divergence algorithm can be used to determine the KL divergence value based on the global sample features and key frame features corresponding to each sample key frame. Among them, the calculation formula of the KL divergence value can be expressed by the following formula (4):

[0118]

[0119] Among them, P c (j) represents the probability value of the global sample feature corresponding to the j-th sample key frame in the probability distribution after normalization of all sample key frame corresponding global sample features f C in the first sample video, P align (j) represents the probability value of the key frame feature corresponding to the j-th sample key frame in the probability distribution after normalization of all sample key frame corresponding key frame features f align in the first sample video, where, ∑ k f align (k) represents the sum of the key frame features corresponding to all sample key frames (i.e., k frames) in the first sample video; ∑ k f C (k) represents the sum of the global sample features corresponding to all sample key frames (i.e., k frames) in the first sample video.

[0120] It can be understood that the smaller the KL divergence value, the closer the distribution between the global sample features corresponding to each sample key frame and the key frame features corresponding to each sample key frame can be characterized. Correspondingly, in step 23, the backbone network and the feature mapping network are trained with the goal of minimizing the difference between the quality prediction results and the quality label results corresponding to each sample key frame, and minimizing the KL divergence value.

[0121] In this step, the electronic device may adopt the foregoing preset loss function, determine the first loss value based on the difference between the quality prediction result and the quality label result corresponding to each sample key frame, and then jointly determine the model loss value based on the first loss value and the KL divergence value, where the model loss value is positively correlated with the first loss value and positively correlated with the KL divergence value; with the goal of minimizing the model loss value, that is, minimizing the difference between the quality prediction result and the quality label result corresponding to each sample key frame, and minimizing the KL divergence value as the goal, train the backbone network and the feature mapping network.

[0122] In the above process, based on the similarity between each frame block in the first key frame and the first key frame, the third segment corresponding to the first key frame is screened from the patch blocks of its corresponding first segment and second segment respectively. Among the third segments corresponding to each key frame obtained in this way, the continuity and integrity of the patch blocks included therein in the spatial dimension with respect to the corresponding video frame are not damaged, and the patch blocks screened from the second segment included in the third segment can contain more information in the original video frame block, providing more useful information for the subsequent evaluation of the video quality. Using such a third segment through the backbone network and the quality prediction network, a quality evaluation result with better accuracy for the first video can be obtained, so as to realize a relatively accurate evaluation of the quality of the first video.

[0123] Moreover, using the output of the specified video scoring network to constrain the global sample features in the output of the feature mapping network in the first branch network, thereby adjusting the parameters of the backbone network and the feature mapping network, so that the distribution of the global sample features in the output of the feature mapping network is closer to the distribution of the key frame features corresponding to each sample key frame encoded by the specified video scoring network; enabling the features obtained by processing through the first branch network to contain more features related to the video quality, providing a basis for obtaining a quality evaluation result with higher accuracy subsequently, so as to better improve the accuracy of the quality evaluation result.

[0124] In some possible examples, as Figure 1 shown, the video quality evaluation model may include the foregoing backbone network, the foregoing first branch network, the second branch network, and the foregoing quality prediction network, where the second branch network may be a network for extracting temporal dimension information in the video, so as to enhance the features extracted by the backbone network using the extracted temporal dimension information, and further obtain features that contribute to improving the accuracy of the quality evaluation result.

[0125] As Figure 4 shown, another method for training a video quality evaluation model provided in the embodiments of this specification may include the following steps S410-S470:

[0126] In step S410, obtain the quality label result corresponding to the sample video, the first sample segment set, and the second sample segment set. Among them, the first sample segment set is obtained by segment sampling of the sample video, and the second sample segment set is obtained by performing the segment sampling on the downsampled sample video obtained based on a specified sampling method, where the specified sampling method is a sampling method based on weighted average.

[0127] In step S420, determine the sample key frame sequence based on the sample video. Among them, there is a corresponding relationship between each sample key frame in the sample key frame sequence and each first sample segment and each second sample segment.

[0128] In step S430, for any first key frame among each key frame, based on the similarity between each frame block in the first key frame and the first key frame, filter out the third segment corresponding to the first key frame from the patch blocks of its corresponding first segment and second segment respectively. The patch blocks of the corresponding first segment and second segment of the first key frame respectively have a corresponding relationship with each frame block in the first key frame.

[0129] Among them, the implementation principle of steps S410 - S430 is similar to the implementation principle of the foregoing steps S210 - S230, and the implementation process can refer to the implementation process of the foregoing steps S210 - S230, which will not be elaborated here.

[0130] In step S440, based on the sample video, determine the sample time feature corresponding to the sample video through the second branch network. Among them, the second branch network can be any network in the related art that can extract time features from the video.

[0131] In some possible examples, in step S440, the following steps 31 - 32 can be included:

[0132] In step 31, sample the sample video to obtain the sample sampling frame sequence. In this step, the sample video can be sampled based on a specified sampling frame rate to obtain the sample sampling frame sequence.

[0133] Then, in step 32, based on the sample sampling frame sequence, determine the sample time feature through the second branch network. In this step, the sample sampling frame sequence can be input into the second branch network to extract the time - dimension features of the sample sampling frame sequence through the second branch network, so as to obtain the sample time feature corresponding to the sample video.

[0134] In some specific examples, the second branch network can be a network based on a dual-path temporal modeling framework, such as the SlowFast network. In the SlowFast network, the "Slow" path can capture the low-frequency global temporal information of the video, that is, it can learn spatial semantic information with video frames of fewer frames and a larger number of channels (for example, the number of channels is C), while the "Fast" path can capture the high-frequency local temporal changes of the video, that is, it can learn motion information with video frames of more frames and a smaller number of channels (for example, the number of channels is βC, where β is less than 1).

[0135] In the case where the second branch network is a network based on a dual-path temporal modeling framework, the aforementioned sampled frame sequence can include a third frame sequence and a fourth frame sequence, where the sampling frame rates corresponding to the third frame sequence and the fourth frame sequence are different. The second branch network can include multiple network layers, hereinafter referred to as branch network layers. Each branch network layer includes a first sub-layer and a second sub-layer, and the multiple branch network layers are arranged in series. The number of layers of the branch network layer of the second branch network can be set according to the task requirements. As Figure 5 shown, the second branch network can include three layers of branch network layers. The first sub-layers in each branch network layer form one processing path, and the second sub-layers in each branch network form another processing path.

[0136] In some cases, the number of sampled video frames in the third frame sequence can be greater than the number of sampled video frames in the fourth frame sequence. The processing path formed by the first sub-layers in each branch network layer can be understood as the aforementioned "Fast" path, and the processing path formed by the second sub-layers in each branch network layer can be understood as the aforementioned "Slow" path.

[0137] Correspondingly, in step 32, it can be specifically set as: through each branch network layer, process the third frame sequence and the fourth frame sequence to obtain the sample temporal features corresponding to the sample video. The data processing methods of each branch network layer are similar. Hereinafter, taking any m-th branch network layer as an example, the data processing process of each branch network layer will be introduced. m is a positive integer, and its maximum value is the number of layers of the branch network layer. The processing process of the m-th branch network layer can include the following steps 321-323:

[0138] In step 321, input the result corresponding to the first sub-layer in the (m - 1)-th branch network layer into the first sub-layer of the m-th branch network layer to obtain a third sub-feature as the result corresponding to the first sub-layer of the m-th branch network layer. When m is equal to 1, the result corresponding to the first sub-layer in the (m - 1)-th branch network layer is the third frame sequence.

[0139] It can be understood that when the m-th branch network layer is not the last layer of the second branch network, the result corresponding to the first sub-layer of the m-th branch network layer is used as the input of the first sub-layer of the (m + 1)-th branch network layer. When the m-th branch network layer is not the last layer of the second branch network, the result corresponding to the first sub-layer of the m-th branch network layer is used to fuse with the result corresponding to the first sub-layer of the m-th branch network layer to obtain the output of the second branch network.

[0140] In step 322, the result corresponding to the second sub-layer in the (m - 1)-th branch network layer is input into the second sub-layer of the m-th branch network layer to obtain a fourth sub-feature. When m equals 1, the result corresponding to the second sub-layer in the (m - 1)-th branch network layer is the fourth frame sequence.

[0141] Among them, when the (m - 1)-th branch network layer is not the first layer of the second branch network, the result corresponding to the second sub-layer in the (m - 1)-th branch network layer is the result obtained by fusing the output of the second sub-layer in the (m - 2)-th branch network layer and the output of the first sub-layer in the (m - 2)-th branch network layer. When the (m - 1)-th branch network layer is the first layer of the second branch network, that is, when m equals 1, the result corresponding to the second sub-layer in the (m - 1)-th branch network layer is the fourth frame sequence.

[0142] After obtaining the fourth sub-feature, in step 323, the third sub-feature and the fourth sub-feature are fused to obtain a sample fusion sub-feature, which is used as the result corresponding to the second sub-layer of the m-th branch network layer.

[0143] In some possible examples, in step 323, the third sub-feature and the fourth sub-feature can be concatenated to obtain a concatenated sub-feature, and then the concatenated sub-feature is processed by a convolutional layer to map the size of the concatenated sub-feature to the size of the fourth sub-feature to obtain a sample fusion sub-feature, which is used as the result corresponding to the second sub-layer of the m-th branch network layer. In some other possible examples, the third sub-feature and the fourth sub-feature can be superimposed to obtain a sample fusion sub-feature, which is used as the result corresponding to the second sub-layer of the m-th branch network layer.

[0144] Furthermore, when the m-th branch network layer is not the last layer, the aforementioned sample fusion sub-feature is used as the input corresponding to the second sub-layer of the subsequent (m + 1)-th branch network layer. When the m-th branch network layer is the last layer, the sample fusion sub-feature is used as the output of the second branch network.

[0145] In some examples, there may be differences in the sizes between the third sub-feature and the fourth sub-feature. Correspondingly, before fusing the two, the sizes of the third sub-feature and the fourth sub-feature can be unified first, and then the third sub-feature and the fourth sub-feature with unified sizes are fused. In some cases, the size of the third sub-feature can be adjusted to the size of the fourth sub-feature.

[0146] In the above manner, the outputs of each branch network layer of the second branch network can be obtained, including the results corresponding to the first sub-layer and the results corresponding to the second sub-layer of each branch network layer. After that, all or part of the results corresponding to the second sub-layer of each branch network layer in the second branch network (including the aforementioned sample fusion sub-features) can be classified into the sample time features for enhancing the features extracted by the backbone network.

[0147] In step S450, by combining each third sample segment and the sample time features, the target video features are obtained through the backbone network.

[0148] In some possible examples, the backbone network may include a plurality of serially arranged network layers, hereinafter referred to as backbone network layers; the plurality of serially arranged backbone network layers of the backbone network may include the first backbone network layer corresponding to the m-th branch network layer; correspondingly, in step S450, the following steps 41-43 may be included:

[0149] In step 41, at least based on each third sample segment, through the first backbone network layer in the backbone network and several backbone network layers before it, the sample intermediate features corresponding to the first backbone network layer are obtained.

[0150] In some embodiments, the electronic device may form the third sample segments into a segment sequence and input it into the backbone network to process the segment sequence through the first backbone network layer in the backbone network and several backbone network layers before it, so as to obtain the sample intermediate features corresponding to the first backbone network layer.

[0151] In some cases, when the first backbone network layer is the first network layer of the backbone network, based on each third sample segment, through the first backbone network layer, the sample intermediate features corresponding to the first backbone network layer are obtained.

[0152] In still other embodiments, at least one backbone network layer before the first backbone network layer in the backbone network may respectively have a corresponding relationship with at least one branch network layer before the m-th branch network layer in the second branch network. Correspondingly, based on each third sample segment and the results corresponding to the second sub-layers of at least one branch network layer before the m-th branch network layer respectively, through the first backbone network layer in the backbone network and several backbone network layers before it, the sample intermediate features corresponding to the first backbone network layer are obtained.

[0153] As Figure 6 shown, assuming that the second branch network includes 3 layers of branch network layers, the backbone network may include: a plurality of backbone network layers respectively corresponding to each layer of branch network layers, as Figure 6As shown, it is assumed that the backbone network includes a backbone network layer a corresponding to the first branch network layer, a backbone network layer b corresponding to the second branch network layer, and a backbone network layer c corresponding to the third branch network layer.

[0154] Correspondingly, it is assumed that the aforementioned backbone network layer b is the first backbone network layer corresponding to the m-th branch network layer, and it is assumed that the backbone network layer a is the first backbone network layer. There is no other backbone network layer between the backbone network layer b and the backbone network layer a. The process of the sample intermediate feature corresponding to the aforementioned first backbone network layer may include:

[0155] Form a fragment sequence by combining each third sample fragment, and input the fragment sequence into the backbone network layer a of the backbone network to process the fragment sequence through the backbone network layer a to obtain the output feature a corresponding to the backbone network layer a; use the specified fusion algorithm mentioned later to obtain the fused feature corresponding to the backbone network layer a based on the output feature a and the result corresponding to the second sub-layer in the first branch network layer; then, input the fused feature corresponding to the backbone network layer a into the backbone network layer b to process the fused feature corresponding to the backbone network layer a through the backbone network layer b to obtain the output feature corresponding to the backbone network layer b, that is, obtain the sample intermediate feature corresponding to the first backbone network layer.

[0156] Figure 6 The structure of the backbone network shown in only shows the backbone network layer a, the backbone network layer b, and the backbone network layer c. In some possible implementation manners, before the backbone network layer a, between the backbone network layer a and the backbone network layer b, between the backbone network layer b and the backbone network layer c, and / or after the backbone network layer c, the backbone network may further include other backbone network layers.

[0157] After obtaining the sample intermediate feature corresponding to the first backbone network layer, in step 42, use the specified fusion algorithm to obtain the sample fusion feature based on the aforementioned sample fusion sub-feature and the sample intermediate feature.

[0158] Among them, the specified fusion algorithm may be any algorithm in the related art that can achieve feature fusion, such as but not limited to algorithms such as superposition, splicing, and pixel-by-pixel multiplication.

[0159] In some possible examples, the specified fusion algorithm may be a fusion algorithm based on the cross-attention mechanism. Correspondingly, the process of obtaining the sample fusion feature in step 42 may include the following steps 421-423:

[0160] In step 421, use the aforementioned sample fusion sub-feature as the sample query matrix. In this step, use the sample fusion sub-feature corresponding to the m-th branch network layer corresponding to the first backbone network layer as the sample query matrix.

[0161] In step 422, the aforementioned sample intermediate features are respectively used as the sample key matrix and the sample value matrix. In this step, the sample intermediate features corresponding to the first backbone network layer are respectively used as the sample key matrix and the sample value matrix.

[0162] Next, in step 423, using the cross-attention mechanism, based on the aforementioned sample query matrix, sample key matrix, and sample value matrix, a sample fusion feature is obtained.

[0163] Among them, the sample fusion feature can be represented by the following formula (5):

[0164]

[0165] Among them, V represents the sample value matrix, that is, the sample intermediate feature corresponding to the first backbone network layer, K represents the sample key matrix, that is, the sample intermediate feature corresponding to the first backbone network layer, Q represents the sample query matrix, that is, the sample fusion sub-feature corresponding to the m-th branch network layer corresponding to the first backbone network layer, d k represents the dimension number of the sample key matrix, softmax(·) represents the activation function, and f fast represents the sample fusion feature corresponding to the first backbone network layer.

[0166] In step 43, based at least on the sample fusion feature, a target video feature is obtained.

[0167] In some possible examples, when the first backbone network layer is the last layer of the backbone network, the sample fusion feature can be used as the target video feature.

[0168] In some other possible examples, when the first backbone network layer is not the last layer of the backbone network, based at least on the sample fusion feature, through several backbone network layers after the first backbone network layer in the backbone network, a target video feature is obtained.

[0169] Among them, when there is no backbone network layer corresponding to the branch network layer of the second branch network among several backbone network layers after the first backbone network layer, the sample fusion feature corresponding to the first backbone network layer can be input into several backbone network layers after the first backbone network layer in the backbone network to obtain the target video feature.

[0170] In the case that there is a backbone network layer corresponding to the branch network layer of the second branch network (subsequently referred to as the second backbone network layer) among several backbone network layers after the first backbone network layer, based on the sample fusion feature and the result corresponding to the second sublayer of the branch network layer corresponding to the second backbone network layer, the target video feature is obtained through several backbone network layers after the first backbone network layer. Among them, for the implementation process of obtaining the target video feature in this case, reference can be made to the process of obtaining the sample intermediate feature corresponding to the first backbone network layer through the first backbone network layer and several backbone network layers before it in the backbone network based on each third sample segment and the results corresponding to the second sublayers of at least one branch network layer before the m-th branch network layer, which will not be elaborated here.

[0171] In some other possible examples, the aforementioned sample time feature may only include the output of the second branch network, that is, the result corresponding to the second sublayer in the last branch network layer. Correspondingly, the electronic device can obtain the output of the backbone network based on each third sample segment through the backbone network. Then, based on the aforementioned specified fusion algorithm, the output of the backbone network and the sample time feature are fused to obtain the target video feature.

[0172] After obtaining the target video feature, in step S460, based on the target video feature, the quality prediction result corresponding to the sample video is obtained through the quality prediction network.

[0173] In step S470, based on the difference between the quality prediction result corresponding to the sample video and the quality label result, the backbone network is trained.

[0174] Among them, the implementation principle of steps S460 - S470 is similar to that of the aforementioned steps S250 - S260, and the implementation process can refer to the implementation process of the aforementioned steps S250 - S260, which will not be elaborated here.

[0175] In the above process, based on the similarity between each frame block in the first sample key frame and the first sample key frame, on the basis of screening out the third sample segment corresponding to the first sample key frame from the patch blocks of its corresponding first sample segment and second sample segment respectively, by combining the sample time feature provided by the second branch network and each third sample segment, the target video feature is obtained through the backbone network, that is, the features in each third sample segment are enhanced through the sample time feature, and the target video feature that can contain more features beneficial to video quality evaluation is obtained, providing a basis for improving the accuracy of the model's video quality evaluation result.

[0176] As Figure 7 shown, an embodiment of this specification provides another method for training a video quality evaluation model, which may include the following steps S710 - S7130:

[0177] In step S710, obtain the quality label result corresponding to the sample video, the first sample segment set, and the second sample segment set. Among them, the first sample segment set is obtained by segment sampling the sample video, and the second sample segment set is obtained by segment sampling the downsampled sample video obtained based on a specified sampling method, and the specified sampling method is a sampling method based on weighted average.

[0178] In step S720, based on the sample video, determine the sample key frame sequence. Among them, there is a corresponding relationship between each sample key frame in the sample key frame sequence and each first sample segment and each second sample segment.

[0179] Among them, the implementation principles of steps S710 - S720 are similar to those of the foregoing steps S210 - S220, and the implementation process can refer to the implementation process of the foregoing steps S210 - S220, which will not be elaborated here.

[0180] In step S730, for any first sample key frame among the sample key frames, based on the first sample key frame, obtain the sample coding feature through the visual coding network.

[0181] In step S740, based on the sample coding feature, obtain the sample mapping feature through the feature mapping network.

[0182] In step S750, based on the sample mapping feature, determine the sample global feature corresponding to the first sample key frame and the sample local features corresponding to each frame block therein.

[0183] In step S760, determine the similarity between each sample local feature corresponding to each frame block in the first sample key frame and its sample global feature respectively.

[0184] In step S770, from the second sample segment corresponding to the first sample key frame, determine M patch blocks corresponding to the M sample local features with the greatest corresponding similarity; and from the first sample segment corresponding to the first sample key frame, determine each patch block corresponding to the other sample local features respectively, so as to obtain the third sample segment corresponding to the first sample key frame.

[0185] Among them, the implementation principles of steps S730 - S770 are similar to those of the foregoing steps 11 - 12 and step S230 respectively, and the implementation process can refer to the implementation process of the foregoing steps 11 - 12 and step S230 respectively, which will not be elaborated here.

[0186] In step S780, based on the sample video, determine the sample time feature corresponding to the sample video through the second branch network.

[0187] In step S790, the target video features are obtained by combining each third sample segment and the sample time features through the backbone network.

[0188] In step S7100, based on the target video features, the quality prediction result corresponding to the sample video is obtained through the quality prediction network.

[0189] Among them, the implementation principle of steps S780 - S7100 is similar to that of the foregoing steps S440 - S460, and the implementation process can refer to the implementation process of the foregoing steps S440 - S460, which will not be elaborated here.

[0190] In step S7110, based on the sample key frame sequence, the key frame features corresponding to each sample key frame are obtained through the specified video scoring network.

[0191] In step S7120, the KL divergence algorithm is adopted to determine the KL divergence value based on the global sample features and key frame features corresponding to each sample key frame.

[0192] In step S7130, with the goal of minimizing the difference between the quality prediction result and the quality label result corresponding to each sample key frame, and minimizing the KL divergence value, the backbone network and the feature mapping network are trained.

[0193] Among them, the implementation principle of steps S7110 - S7130 is similar to that of the foregoing steps 21 - 23, and the implementation process can refer to the implementation process of the foregoing steps 21 - 23, which will not be elaborated here.

[0194] In the above process, the sample time features provided by the second branch network and each third sample segment containing more information beneficial to video quality evaluation are combined, and the target video features are obtained through the backbone network, that is, the features in each third sample segment are enhanced by the sample time features to obtain the target video features that can contain more features beneficial to video quality evaluation, providing a basis for improving the accuracy of the model's video quality evaluation result, and using the output of the specified video scoring network to constrain the global sample features in the output of the first branch network, jointly adjusting the parameters of the backbone network and the feature mapping network, so that the first branch network can extract more information beneficial to the video quality evaluation result from the sample key frame sequence, and further providing a basis for improving the accuracy of the model's video quality evaluation result to improve the accuracy of the output result of the video quality evaluation model.

[0195] Train the video quality evaluation model in the above manner until the preset convergence condition is reached, determine that the training of the video quality evaluation model is completed, and then the trained video quality evaluation model can be used to evaluate the video quality. Among them, the preset convergence condition may include, but is not limited to: the number of iterations exceeds the specified number threshold, the training duration exceeds the specified duration, or the aforementioned model loss value is lower than the specified threshold.

[0196] The process of evaluating the video quality is introduced below.

[0197] Corresponding to the above method embodiments, the embodiments of this specification provide a method for evaluating video quality, as Figure 8 shown, the method includes steps S810 - S850:

[0198] In step S810, obtain a first segment set and a second segment set. Among them, the first segment set is obtained by segment sampling of a first video, and the second segment set is obtained by performing the segment sampling on the downsampled first video obtained based on a specified sampling method, and the specified sampling method is a sampling method based on weighted average.

[0199] Among them, the first video can be any video that needs to be evaluated for quality. The implementation principle of step S810 is similar to that of step S210 described above, and its implementation process can refer to the implementation process of step S210 described above, which will not be elaborated here.

[0200] In step S820, based on the first video, determine a key frame sequence, where each key frame in the key frame sequence has a corresponding relationship with each first segment and each second segment. Among them, the implementation principle of step S820 is similar to that of step S220 described above, and its implementation process can refer to the implementation process of step S220 described above, which will not be elaborated here.

[0201] In step S830, for any first key frame among each key frame, based on the similarity between each frame block in the first key frame and the first key frame itself, select the third segment corresponding to the first key frame from the patch blocks of its corresponding first segment and second segment respectively. The patch blocks of the corresponding first segment and second segment of the first key frame respectively have a corresponding relationship with each frame block in the first key frame. Among them, the implementation principle of step S830 is similar to that of step S230 described above, and its implementation process can refer to the implementation process of step S230 described above, which will not be elaborated here.

[0202] In step S840, determine the first video feature based on the third segment corresponding to each key frame.

[0203] In some possible examples, in step S840, it may include: determining a first video feature based on the third segment corresponding to each key frame through the backbone network in the aforementioned video quality evaluation model that has completed training. Among them, the implementation principle of step S840 is similar to that of the aforementioned step S240, and its implementation process can refer to the implementation process of the aforementioned step S240, which will not be elaborated here.

[0204] In still some other possible examples, in step S840, a traditional feature extraction method may be used to extract features from the third segment corresponding to each key frame to obtain a first video feature.

[0205] In step S850, a quality evaluation result corresponding to the first video is obtained based on the first video feature.

[0206] In some possible examples, in step S850, it may include: obtaining a quality evaluation result corresponding to the first video based on the first video feature through the quality prediction network in the video quality evaluation model. Among them, the implementation principle of step S850 is similar to that of the aforementioned step S250, and its implementation process can refer to the implementation process of the aforementioned step S250, which will not be elaborated here.

[0207] In some possible examples, the video quality evaluation model may further include a first branch network;

[0208] Before step S830, it further includes: determining the global visual feature corresponding to the first key frame and the local visual features corresponding to each frame block therein through the first branch network based on the first key frame; determining the similarity between the local visual features corresponding to each frame block in the first key frame and its global visual feature respectively;

[0209] In step S830, it includes: determining M patch blocks corresponding to the M local visual features with the greatest similarity from the second segment corresponding to the first key frame; and determining the patch blocks corresponding to the other local visual features respectively from the first segment corresponding to the first key frame to obtain the third segment corresponding to the first key frame.

[0210] In some possible examples, the first branch network may include a visual coding network and a feature mapping network. Among them, the feature mapping network is trained by constraining the output features obtained by processing a specified video frame by the feature mapping network with the output features obtained by processing the specified video frame by a specified video scoring network;

[0211] The above-mentioned determination of the global visual feature corresponding to the first key frame and the local visual features corresponding to each frame block therein includes: based on the first key frame, obtaining a first encoded feature through the visual encoding network;

[0212] Based on the first encoded feature, obtaining a first mapped feature through the feature mapping network;

[0213] Based on the first mapped feature, determining the global visual feature corresponding to the first key frame and the local visual features corresponding to each frame block therein.

[0214] In some possible examples, the specified video frame may include: a sample key frame corresponding to a sample video for training the video quality evaluation model.

[0215] In some possible examples, the video quality evaluation model may further include a second branch network;

[0216] Before step S840, it further includes: based on the first video, determining the time feature corresponding to the first video through the second branch network;

[0217] In step S840, it includes: combining each third segment and the time feature, and obtaining the first video feature through the backbone network.

[0218] In some possible examples, the above-mentioned determination of the time feature corresponding to the first video includes:

[0219] Sampling the first video to obtain a sampling frame sequence; based on the sampling frame sequence, determining the time feature through the second branch network.

[0220] In some possible examples, the sampling frame sequence includes a first frame sequence and a second frame sequence, wherein the sampling frame rates corresponding to the first frame sequence and the second frame sequence are different; the second branch network includes multiple branch network layers, and each branch network layer includes a first sub-layer and a second sub-layer;

[0221] The above-mentioned determination of the time feature includes: processing the first frame sequence and the second frame sequence through each branch network layer to obtain the time feature corresponding to the first video, wherein the processing process of the m-th branch network layer includes:

[0222] Inputting the result corresponding to the first sub-layer in the (m - 1)-th branch network layer into the first sub-layer of the m-th branch network layer to obtain a first sub-feature as the result corresponding to the first sub-layer of the m-th branch network layer, wherein when m is equal to 1, the result corresponding to the first sub-layer in the (m - 1)-th branch network layer is the first frame sequence;

[0223] Input the result corresponding to the second sub-layer in the (m - 1)-th branch network layer into the second sub-layer of the m-th branch network layer to obtain a second sub-feature. When m equals 1, the result corresponding to the second sub-layer in the (m - 1)-th branch network layer is the second frame sequence;

[0224] Fuse the first sub-feature and the second sub-feature to obtain a fused sub-feature, which serves as the result corresponding to the second sub-layer of the m-th branch network layer.

[0225] In some possible examples, the backbone network includes a first backbone network layer corresponding to the m-th branch network layer;

[0226] In step S840, it includes: Based at least on each third segment, through the first backbone network layer in the backbone network and several backbone network layers before it, obtain the intermediate feature corresponding to the first backbone network layer;

[0227] Adopt a specified fusion algorithm to obtain a fusion feature based on the fused sub-feature and the intermediate feature;

[0228] Based at least on the fusion feature, obtain the first video feature.

[0229] In some possible examples, the foregoing obtaining of the fusion feature includes: Using the fused sub-feature as a query matrix; using the intermediate feature as a key matrix and a value matrix respectively; adopting a cross-attention mechanism to obtain the fusion feature based on the query matrix, the key matrix, and the value matrix.

[0230] The above process can obtain a video quality evaluation result with relatively higher accuracy through a video quality evaluation model, which can provide a better video evaluation function for a short video platform or other video platforms. It helps the short video platform or other video platforms recommend videos with better quality to users and improve the customer retention rate of the short video platform or other video platforms.

[0231] The above content describes specific embodiments of this specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order than in the embodiments, and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily have to be executed in the specific order or continuous order shown to achieve the desired result. In certain embodiments, multi-tasking and parallel processing are also possible, or may be advantageous.

[0232] Corresponding to the above method embodiment, an embodiment of this specification provides a video quality evaluation device 900, the schematic block diagram of which is as Figure 9 shown, including:

[0233] The first acquisition module 910 is configured to acquire a first segment set and a second segment set, where the first segment set is obtained by segment sampling of a first video, and the second segment set is obtained by segment sampling of the downsampled first video obtained based on a specified sampling method, and the specified sampling method is a sampling method based on weighted average;

[0234] The first determination module 920 is configured to determine a key frame sequence based on the first video, where there is a corresponding relationship between each key frame in the key frame sequence and each first segment and each second segment;

[0235] The first screening module 930 is configured to, for any first key frame among each key frame, based on the similarity between each frame block in the first key frame and the first key frame, screen and obtain a third segment corresponding to the first key frame from the patch blocks of its corresponding first segment and second segment respectively, and the patch blocks of the corresponding first segment and second segment of the first key frame corresponding to the first key frame respectively have a corresponding relationship with each frame block in the first key frame;

[0236] The second determination module 940 is configured to determine a first video feature based on the third segment corresponding to each key frame;

[0237] The first obtaining module 950 is configured to obtain a quality evaluation result corresponding to the first video based on the first video feature.

[0238] In some possible examples, the video quality evaluation model further includes a first branch network; the apparatus further includes:

[0239] The fifth determination module (not shown in the figure) is configured to, before screening and obtaining the third segment corresponding to the first key frame, determine the global visual feature corresponding to the first key frame and the local visual features corresponding to each frame block therein through the first branch network based on the first key frame;

[0240] The sixth determination module (not shown in the figure) is configured to determine the similarity between the local visual features corresponding to each frame block in the first key frame and its global visual feature respectively;

[0241] The first screening module 930 is specifically configured to determine M patch blocks corresponding to the M local visual features with the greatest similarity from the second segment corresponding to the first key frame; and determine the patch blocks corresponding to the other local visual features from the first segment corresponding to the first key frame, so as to obtain the third segment corresponding to the first key frame.

[0242] In some possible examples, the first branch network includes a visual encoding network and a feature mapping network. Among them, the feature mapping network is trained by constraining the output features obtained by processing a specified video frame using a specified video scoring network with the output features obtained by processing the specified video frame by the feature mapping network.

[0243] The sixth determination module is specifically configured to, based on the first key frame, obtain a first encoded feature through the visual encoding network; based on the first encoded feature, obtain a first mapped feature through the feature mapping network; and based on the first mapped feature, determine the global visual feature corresponding to the first key frame and the local visual features corresponding to each frame block therein.

[0244] In some possible examples, the specified video frame includes: a sample key frame corresponding to a sample video for training the video quality evaluation model.

[0245] In some possible examples, the video quality evaluation model further includes a second branch network; the apparatus further includes:

[0246] A seventh determination module (not shown in the figure), configured to, before determining the first video feature, based on the first video, determine the temporal feature corresponding to the first video through the second branch network;

[0247] The second determination module 940 is specifically configured to jointly use each third segment and the temporal feature to obtain the first video feature through the backbone network.

[0248] In some possible examples, the seventh determination module is specifically configured to sample the first video to obtain a sampled frame sequence; and based on the sampled frame sequence, determine the temporal feature through the second branch network.

[0249] In some possible examples, the sampled frame sequence includes a first frame sequence and a second frame sequence, where the sampling frame rates corresponding to the first frame sequence and the second frame sequence are different; the second branch network includes multiple branch network layers, and each branch network layer includes a first sub-layer and a second sub-layer;

[0250] The seventh determination module is specifically configured to process the first frame sequence and the second frame sequence through each branch network layer to obtain the temporal feature corresponding to the first video, where the processing process of the m-th branch network layer includes:

[0251] Input the result corresponding to the first sub - layer in the (m - 1)-th branch network layer into the first sub - layer of the m - th branch network layer to obtain the first sub - feature, which is used as the result corresponding to the first sub - layer of the m - th branch network layer. When m equals 1, the result corresponding to the first sub - layer in the (m - 1)-th branch network layer is the first frame sequence;

[0252] Input the result corresponding to the second sub - layer in the (m - 1)-th branch network layer into the second sub - layer of the m - th branch network layer to obtain the second sub - feature. When m equals 1, the result corresponding to the second sub - layer in the (m - 1)-th branch network layer is the second frame sequence;

[0253] Fuse the first sub - feature and the second sub - feature to obtain the fused sub - feature, which is used as the result corresponding to the second sub - layer of the m - th branch network layer.

[0254] In some possible examples, the backbone network includes the first backbone network layer corresponding to the m - th branch network layer;

[0255] The second determination module 940 is specifically configured to, at least based on each third segment, obtain the intermediate feature corresponding to the first backbone network layer through the first backbone network layer and several backbone network layers before it in the backbone network; use a specified fusion algorithm to obtain a fusion feature based on the fused sub - feature and the intermediate feature; and obtain the first video feature at least based on the fusion feature.

[0256] In some possible examples, the second determination module 940 is specifically configured to use the fused sub - feature as a query matrix; use the intermediate feature as a key matrix and a value matrix respectively; and use a cross - attention mechanism to obtain the fusion feature based on the query matrix, the key matrix, and the value matrix.

[0257] Corresponding to the above - mentioned method embodiments, an embodiment of this specification provides an apparatus 1000 for training a video quality evaluation model. The video quality evaluation model includes a backbone network and a quality prediction network, and its schematic block diagram is as Figure 10 shown, including:

[0258] A second acquisition module 1010 is configured to acquire the quality label result corresponding to the sample video, a first sample segment set, and a second sample segment set. The first sample segment set is obtained by segment sampling of the sample video, and the second sample segment set is obtained by segment sampling of the down - sampled sample video obtained based on a specified sampling method. The specified sampling method is a sampling method based on weighted average;

[0259] The third determination module 1020 is configured to determine a sample key frame sequence based on the sample video, where there is a corresponding relationship between each sample key frame in the sample key frame sequence and each first sample segment and each second sample segment;

[0260] The second screening module 1030 is configured to, for any first sample key frame among each sample key frame, based on the similarity between each frame block in the first sample key frame and the first sample key frame, screen out the third sample segment corresponding to the first sample key frame from the patch blocks of its corresponding first sample segment and second sample segment respectively. The patch blocks of the corresponding first sample segment and second sample segment of the first sample key frame respectively have a corresponding relationship with each frame block in the first sample key frame;

[0261] The fourth determination module 1040 is configured to determine target video features through the backbone network based on the third sample segment corresponding to each sample key frame;

[0262] The second obtaining module 1050 is configured to obtain a quality prediction result corresponding to the sample video through the quality prediction network based on the target video features;

[0263] The training module 1060 is configured to train the backbone network based on the difference between the quality prediction result and the quality label result;

[0264] In some possible examples, the video quality evaluation model further includes a first branch network; the apparatus further includes:

[0265] An eighth determination module (not shown in the figure) is configured to, before screening out the third sample segment corresponding to the first sample key frame, determine the global visual feature corresponding to the first sample key frame and the local visual features corresponding to each frame block therein through the first branch network based on the first sample key frame;

[0266] A ninth determination module (not shown in the figure) is configured to determine the similarity between the local visual features corresponding to each frame block in the first sample key frame and its global visual feature respectively;

[0267] The second screening module 1030 is specifically configured to determine M patch blocks corresponding to the M local visual features with the greatest similarity from the second sample segment corresponding to the first sample key frame; and determine the patch blocks corresponding to the other local visual features from the first sample segment corresponding to the first sample key frame, so as to obtain the third sample segment corresponding to the first sample key frame.

[0268] In some possible examples, the first branch network includes a visual coding network and a feature mapping network;

[0269] The ninth determination module is specifically configured to obtain sample encoded features based on the first sample key frame through the visual encoding network;

[0270] obtain sample mapped features based on the sample encoded features through the feature mapping network;

[0271] determine the global sample features corresponding to the first sample key frame and the local sample features corresponding to each frame block therein based on the sample mapped features.

[0272] In some possible examples, the training module 1060 is specifically configured to obtain key frame features corresponding to each sample key frame based on the sample key frame sequence through a specified video scoring network; use the KL divergence algorithm to determine the KL divergence value based on the global sample features and key frame features corresponding to each sample key frame; and train the backbone network and the feature mapping network with the goal of minimizing the difference between the quality prediction results and quality label results corresponding to each sample key frame and minimizing the KL divergence value.

[0273] The above device embodiments correspond to the method embodiments. For specific descriptions, reference can be made to the descriptions in the method embodiment part, which will not be elaborated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, reference can be made to the corresponding method embodiments.

[0274] This specification embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute the method for evaluating video quality or the method for training a video quality evaluation model provided in this specification.

[0275] This specification embodiment also provides a computing device, including a memory and a processor. An executable code is stored in the memory. When the processor executes the executable code, the method for evaluating video quality or the method for training a video quality evaluation model provided in this specification is implemented.

[0276] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple. For relevant parts, reference can be made to the partial descriptions in the method embodiments.

[0277] Those skilled in the art should be able to realize that in one or more of the above examples, the functions described in the embodiments of the present invention can be implemented by hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0278] The specific implementation manners described above have further elaborated on the objectives, technical solutions, and beneficial effects of the embodiments of the present invention. It should be understood that the above are only the specific implementation manners of the embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for evaluating video quality, the method comprising: Obtaining a first segment set and a second segment set, wherein the first segment set is obtained by performing segment sampling on the first video, and the second segment set is obtained by performing segment sampling on the first video after downsampling based on a specified sampling method, and the specified sampling method is a sampling method based on weighted average; Based on the first video, determining a key frame sequence, wherein each key frame in the key frame sequence corresponds to each first segment and each second segment respectively; For any first key frame among the key frames, based on the similarity between each frame block in the first key frame and the first key frame, a third segment corresponding to the first key frame is obtained by screening from the patch blocks of the first segment and the second segment corresponding to the first key frame, and the patch blocks of the first segment and the second segment corresponding to the first key frame have a corresponding relationship with each frame block in the first key frame; Determining a first video feature based on a third segment corresponding to each key frame; Based on the first video feature, a quality evaluation result corresponding to the first video is obtained.

2. The method of claim 1, wherein: The determining the first video feature based on the third segment corresponding to each key frame includes: determining the first video feature through a backbone network in a video quality evaluation model based on the third segment corresponding to each key frame; The obtaining of a quality evaluation result corresponding to the first video based on the first video feature includes: obtaining a quality evaluation result corresponding to the first video through a quality prediction network in a video quality evaluation model based on the first video feature.

3. The method of claim 1, wherein: The video quality assessment model also includes a first branch network; Before the screening to obtain the third segment corresponding to the first key frame, the method further includes: Based on the first key frame, determining, through the first branch network, a global visual feature corresponding to the first key frame and a local visual feature corresponding to each frame block therein; Determine the similarity between the local visual features corresponding to each frame block in the first key frame and its global visual features; The screening to obtain a third segment corresponding to the first key frame includes: From the second fragment corresponding to the first key frame, determine M patch blocks corresponding to the M local visual features with the greatest similarity; and from the first fragment corresponding to the first key frame, determine each patch block corresponding to other local visual features to obtain the third fragment corresponding to the first key frame.

4. The method of claim 3, wherein: The first branch network includes a visual encoding network and a feature mapping network, wherein the feature mapping network is trained by constraining the output features obtained by processing the specified video frame using the output features obtained by the specified video scoring network; The determining of the global visual features corresponding to the first key frame and the local visual features corresponding to each frame block therein includes: Based on the first key frame, obtaining a first coding feature through the visual coding network; Based on the first encoding feature, obtaining a first mapping feature through the feature mapping network; Based on the first mapping features, a global visual feature corresponding to the first key frame and a local visual feature corresponding to each frame block therein are determined.

5. The method of claim 4, wherein: The designated video frames include: sample key frames corresponding to the sample videos used to train the video quality assessment model.

6. The method according to any one of claims 1 to 5, wherein: The video quality assessment model also includes a second branch network; Before determining the first video feature, the method further includes: Based on the first video, determining a time feature corresponding to the first video through the second branch network; The determining of the first video feature includes: The first video feature is obtained by combining the third segments and the time feature through the backbone network.

7. The method of claim 6, wherein: The determining the time feature corresponding to the first video includes: Sampling the first video to obtain a sampling frame sequence; Based on the sampling frame sequence, the time feature is determined through the second branch network.

8. The method of claim 7, wherein: The sampling frame sequence includes a first frame sequence and a second frame sequence, wherein the first frame sequence and the second frame sequence respectively correspond to different sampling frame rates; the second branch network includes a plurality of branch network layers, each branch network layer includes a first sublayer and a second sublayer; The determining of the time feature comprises: The first frame sequence and the second frame sequence are processed through each branch network layer to obtain a time feature corresponding to the first video, wherein the processing process of the mth branch network layer includes: Input the result corresponding to the first sublayer in the m-1th branch network layer into the first sublayer of the m-th branch network layer to obtain a first sub-feature as the result corresponding to the first sublayer of the m-th branch network layer, wherein when m is equal to 1, the result corresponding to the first sublayer in the m-1th branch network layer is the first frame sequence; Input the result corresponding to the second sublayer in the m-1th branch network layer into the second sublayer of the m-th branch network layer to obtain a second sub-feature, wherein when m is equal to 1, the result corresponding to the second sublayer in the m-1th branch network layer is the second frame sequence; The first sub-feature and the second sub-feature are fused to obtain a fused sub-feature as a result corresponding to the second sub-layer of the m-th branch network layer.

9. The method of claim 8, wherein: The backbone network includes a first backbone network layer corresponding to the mth branch network layer; The obtaining of the first video feature includes: At least based on each third segment, obtaining intermediate features corresponding to the first backbone network layer through the first backbone network layer and several backbone network layers before the first backbone network layer in the backbone network; Using a specified fusion algorithm, based on the fused sub-features and the intermediate features, to obtain a fused feature; The first video feature is obtained based at least on the fused feature.

10. The method of claim 9, wherein: The obtained fusion features include: Using the fused sub-features as a query matrix; The intermediate features are used as a key matrix and a value matrix respectively; The fused features are obtained based on the query matrix, the key matrix and the value matrix by adopting a cross attention mechanism.

11. A method for training a video quality assessment model, the video quality assessment model comprising: Backbone network and quality prediction network, including: Obtaining a quality label result, a first sample segment set, and a second sample segment set corresponding to a sample video, wherein the first sample segment set is obtained by performing segment sampling on the sample video, and the second sample segment set is obtained by performing segment sampling on a downsampled sample video obtained based on a specified sampling method, and the specified sampling method is a sampling method based on weighted average; Based on the sample video, determining a sample key frame sequence, wherein each sample key frame in the sample key frame sequence has a corresponding relationship with each first sample segment and each second sample segment; For any first sample key frame among the sample key frames, based on the similarity between each frame block in the first sample key frame and the first sample key frame, a third sample segment corresponding to the first sample key frame is screened from the respective patch blocks of the corresponding first sample segment and the second sample segment, wherein the respective patch blocks of the first sample segment and the second sample segment corresponding to the first sample key frame have a corresponding relationship with each frame block in the first sample key frame; Based on the third sample segments corresponding to each sample key frame, determining the target video features through the backbone network; Based on the target video features, obtaining a quality prediction result corresponding to the sample video through the quality prediction network; The backbone network is trained based on the difference between the quality prediction result and the quality label result.

12. The method of claim 11, wherein: The video quality assessment model also includes a first branch network; Before the screening to obtain the third sample segment corresponding to the first sample key frame, the method further includes: Based on the first sample key frame, determining, through the first branch network, a global visual feature corresponding to the first sample key frame and a local visual feature corresponding to each frame block therein; Determine the similarity between the local visual features corresponding to each frame block in the first sample key frame and its global visual features; The screening to obtain a third sample segment corresponding to the first sample key frame includes: From the second sample segment corresponding to the first sample key frame, determine M patch blocks corresponding to the M local visual features with the greatest similarity; and from the first sample segment corresponding to the first sample key frame, determine each patch block corresponding to other local visual features to obtain a third sample segment corresponding to the first sample key frame.

13. The method of claim 12, wherein: The first branch network includes a visual encoding network and a feature mapping network; The determining of the similarity between the local visual features corresponding to each frame block in the first sample key frame and its global visual features respectively comprises: Based on the first sample key frame, obtaining a sample coding feature through the visual coding network; Based on the sample encoding feature, obtaining a sample mapping feature through the feature mapping network; Based on the sample mapping features, a global sample feature corresponding to the first sample key frame and a local sample feature corresponding to each frame block therein are determined.

14. The method of claim 13, wherein: The training of the backbone network comprises: Based on the sample key frame sequence, obtaining key frame features corresponding to each sample key frame through a specified video scoring network; The KL divergence algorithm is used to determine the KL divergence value based on the global sample features and key frame features corresponding to each sample key frame; The backbone network and the feature mapping network are trained with the goal of minimizing the difference between the quality prediction results and the quality label results corresponding to each sample key frame and minimizing the KL divergence value.

15. A device for evaluating video quality, the device comprising: A first acquisition module is configured to acquire a first segment set and a second segment set, wherein the first segment set is obtained by performing segment sampling on the first video, and the second segment set is obtained by performing the segment sampling on the first video after downsampling based on a specified sampling method, and the specified sampling method is a sampling method based on weighted average; A first determining module is configured to determine a key frame sequence based on the first video, wherein each key frame in the key frame sequence corresponds to each first segment and each second segment respectively; A first screening module is configured to screen, for any first key frame among the key frames, from the respective patch blocks of the first segment and the second segment corresponding to the first key frame based on the similarity between each frame block in the first key frame and the first key frame, to obtain a third segment corresponding to the first key frame, wherein the respective patch blocks of the first segment and the second segment corresponding to the first key frame have a corresponding relationship with each frame block in the first key frame; A second determination module is configured to determine the first video feature based on the third segment corresponding to each key frame; The first obtaining module is configured to obtain a quality evaluation result corresponding to the first video based on the first video feature.

16. A device for training a video quality assessment model, the video quality assessment model comprising: Backbone network and quality prediction network, including: A second acquisition module is configured to acquire a quality label result corresponding to the sample video, a first sample segment set, and a second sample segment set, wherein the first sample segment set is obtained by performing segment sampling on the sample video, and the second sample segment set is obtained by performing segment sampling on the sample video after downsampling based on a specified sampling method, and the specified sampling method is a sampling method based on weighted average; A third determination module is configured to determine a sample key frame sequence based on the sample video, wherein each sample key frame in the sample key frame sequence has a corresponding relationship with each first sample segment and each second sample segment respectively; The second screening module is configured to screen, for any first sample key frame among the sample key frames, from the respective patch blocks of the corresponding first sample segment and the second sample segment based on the similarity between each frame block in the first sample key frame and the first sample key frame, to obtain a third sample segment corresponding to the first sample key frame, wherein the respective patch blocks of the first sample segment and the second sample segment corresponding to the first sample key frame have a corresponding relationship with each frame block in the first sample key frame; A fourth determination module is configured to determine the target video feature through the backbone network based on the third sample segments corresponding to each sample key frame; A second obtaining module is configured to obtain a quality prediction result corresponding to the sample video through the quality prediction network based on the target video feature; A training module is configured to train the backbone network based on the difference between the quality prediction result and the quality label result.

17. A computing device comprising a memory and a processor, wherein: The memory stores executable code, and when the processor executes the executable code, the method described in any one of claims 1-10 or 11-14 is implemented.

Images