Video quality prediction method, device, equipment, storage medium and program product

Abstract

The present application relates to a method, device, equipment, storage medium and program product for predicting video quality. The method relates to artificial intelligence technology, and the method includes: obtaining video information of a current video derived from an original video, the current video is obtained by encoding, transmitting and decoding the original video, and the video information of the current video includes the clarity parameter, encoding parameter, current resolution parameter and image quality distortion index of the current video; obtaining video information of the original video, the video information of the original video includes the bit rate, original image quality score and original resolution parameter of the original video; calling a trained video quality assessment model, obtaining a video quality feature representation of the current video according to the video information of the current video and the video information of the original video, and outputting a current image quality score based on the video quality feature representation, the current image quality score is used to characterize the degree of image quality degradation of the current video, and improves the efficiency of video quality prediction.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video quality prediction method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of the Internet, video content has become an important part of Internet traffic. Especially with the rapid development of social media, online video platforms and streaming services, the consumption and influence of video content continue to grow. Therefore, the evaluation of video quality is particularly important.

[0003] In related technologies, image quality assessment usually relies on frame extraction and image understanding of video content. Specifically, a series of video frames are obtained from the video, and then the video frames are processed separately. Image understanding is performed based on the processed video frames to determine the video quality. This method requires reasoning and image understanding for each video frame of the video, which greatly increases the computational burden and reduces the efficiency of video quality assessment. Summary of the invention

[0004] Based on this, it is necessary to provide a video quality prediction method, device, equipment, storage medium and program product that can improve the efficiency of video quality prediction in response to the above technical problems.

[0005] In a first aspect, the present application provides a video quality prediction method, comprising:

[0006] Acquire video information of a current video, where the current video is derived from an original video, and the current video is obtained by encoding, transmitting, and decoding the original video, and the video information of the current video includes a definition parameter, an encoding parameter, a current resolution parameter, and a picture quality distortion index of the current video;

[0007] Acquire video information of the original video, where the video information of the original video includes a bit rate, an original picture quality score, and an original resolution parameter of the original video;

[0008] Call the trained video quality assessment model, obtain the video quality feature representation of the current video according to the video information of the current video and the video information of the original video, and output the current picture quality score of the current video based on the video quality feature representation, wherein the current picture quality score is used to characterize the degree of picture quality degradation of the current video.

[0009] In a second aspect, the present application also provides a video quality prediction device, comprising:

[0010] A video information acquisition module is used to acquire video information of a current video, wherein the current video is derived from an original video, and the current video is obtained by encoding, transmitting, and decoding the original video, and the video information of the current video includes a definition parameter, encoding parameter, current resolution parameter, and image quality distortion index of the current video;

[0011] The video information acquisition module is further used to acquire video information of the original video, where the video information of the original video includes a bit rate, an original picture quality score, and an original resolution parameter of the original video;

[0012] The image quality score acquisition module is used to call the trained video quality assessment model, obtain the video quality feature representation of the current video according to the video information of the current video and the video information of the original video, and output the current image quality score of the current video based on the video quality feature representation, wherein the current image quality score is used to characterize the degree of image quality degradation of the current video.

[0013] In some embodiments, the video information acquisition module is also used to obtain a video quality prediction request, and parse the video quality prediction request to determine the video identifier of the current video for video quality prediction; based on the video identifier, read the corresponding video description file from the local storage, and obtain the video information of the current video and the video information of the original video from which the current video originates from the video description file.

[0014] In some embodiments, the video information acquisition module is also used to receive a video description file of the original video sent by the video sending end and video encoding data obtained by encoding the original video, and parse the video information of the original video from the video description file of the original video; decode the video encoding data to obtain the current video, and count the clarity parameters, encoding parameters, current resolution parameters and image quality distortion indicators of the current video to obtain the video information of the current video; generate a video description file of the current video based on the video information of the original video and the video information of the current video, and store the video description file of the current video in correspondence with the video identifier of the current video.

[0015] In some embodiments, the video information acquisition module is further used to calculate a mean square error based on respective pixel values ​​of the current video and the original video to obtain an image quality distortion index of the current video.

[0016] In some embodiments, the video information acquisition module is also used to input multiple video frames in the original video into a video feature-based image quality assessment model for feature extraction, obtain visual features of each video frame, and determine the original image quality score of the original video based on the visual features of each video frame.

[0017] In some embodiments, the trained video quality assessment model includes multiple feature extraction networks connected in sequence, and the image quality score acquisition module is used to input the video information of the current video and the video information of the original video into the first feature extraction network among the multiple feature extraction networks connected in sequence to perform feature extraction to obtain the first intermediate feature; starting from the second feature extraction network, the intermediate feature output by the previous feature extraction network is extracted through the current feature extraction network to obtain the corresponding intermediate feature, until the last feature extraction network outputs the video quality feature representation of the current video.

[0018] In some embodiments, the device also includes a picture quality score fusion module, which is used to obtain the transmission network parameters corresponding to the original video and the object behavior data about the current video; based on the encoding parameters of the current video, the transmission network parameters and the object behavior data, the video quality of the current video is predicted to obtain a reference picture quality score of the current video; the reference picture quality score of the current video and the current picture quality score are fused to obtain a target picture quality score of the current video, and the target picture quality score is used to characterize the degree of picture quality degradation of the current video.

[0019] In some embodiments, the image quality score fusion module is also used to call an image quality assessment model based on video-related information to respectively determine the image quality scores corresponding to the encoding parameters of the current video, the transmission network parameters, and the object behavior data; and fuse the image quality scores corresponding to the encoding parameters of the current video, the transmission network parameters, and the object behavior data to obtain a reference image quality score for the current video.

[0020] In some embodiments, the device also includes a model training module, which is used to obtain training samples, wherein the training samples include video information of a target sample video and video information of an original sample video, wherein the target sample video is derived from the original sample video, and the target sample video is obtained by encoding, transmitting, and decoding the original sample video, wherein the video information of the target sample video includes clarity parameters, encoding parameters, resolution parameters, and image quality distortion indicators of the target sample video, and the video information of the original sample video includes the bit rate, original image quality score, and resolution parameters of the original sample video; according to the video information of the target sample video and the video information of the original sample video, a video quality feature representation of the target sample video is obtained through a video quality assessment model, and an image quality score of the target sample video is output based on the video quality feature representation; based on the difference between the labeled data of the training sample and the image quality score of the target sample video, the video quality assessment model is trained to obtain a trained video quality assessment model.

[0021] In some embodiments, the model training module is used to input multiple video frames in the target sample video into a video feature-based image quality assessment model for feature extraction to obtain visual features of each video frame, determine the image quality score of the target sample video based on the visual features of each video frame, and determine the annotation data of the training sample based on the image quality score of the target sample video.

[0022] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned video quality prediction method when executing the computer program.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned video quality prediction method are implemented.

[0024] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video quality prediction method when executed by a processor.

[0025] The above-mentioned video quality prediction method, device, equipment, storage medium and program product can know the clarity parameters, encoding parameters, current resolution parameters and image quality distortion indicators of the current video by obtaining the video information of the current video. This process does not require video frame extraction of the current video, nor does it require corresponding processing of multiple video frames. Considering that the current video is obtained by encoding, transmitting and decoding the original video, that is, the current video is derived from the original video, therefore, in order to make a comprehensive and effective estimate of the image quality of the current video, the video information of the original video is obtained to determine the bit rate, original image quality score and original resolution parameters of the original video. Then, the trained video quality assessment model is called, and based on the video information of the current video and the video information of the original video, the video quality feature representation of the current video can be extracted, so as to accurately reflect the image quality stop loss of the current video relative to the original video, and obtain a current image quality score of the current video with high accuracy. The current image quality score is used to characterize the degree of image quality loss of the current video. In this way, in the process of predicting the image quality of the current video, there is no need to perform image understanding on each video frame in the current video, which reduces the computational burden and the amount of calculation, thereby improving the efficiency of video quality prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A diagram showing an application environment of a video quality prediction method in an embodiment;

[0028] Figure 2 is a schematic diagram of a flow chart of a video quality prediction method in an embodiment;

[0029] Figure 3 A schematic flow chart of steps for determining a video description file of a current video in one embodiment;

[0030] Figure 4 is a schematic diagram of a model structure of a video quality assessment model in an embodiment;

[0031] Figure 5 is a schematic diagram of a model structure of a video quality assessment model in another embodiment;

[0032] Figure 6 A schematic diagram of a flow chart of a model training step in an embodiment;

[0033] Figure 7 A schematic diagram of a process for determining a target image quality score in one embodiment;

[0034] Figure 8 is a schematic diagram of video quality prediction in one embodiment;

[0035] Fig. 9 is a schematic diagram of a video quality prediction system architecture in one embodiment;

[0036] Fig.10 is a structural block diagram of a video quality prediction device in one embodiment;

[0037] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] The video quality prediction method provided in the embodiment of the present application involves artificial intelligence (AI) technology. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0040] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, and mechatronics. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The embodiments of this application specifically relate to machine learning technology of artificial intelligence.

[0041] In related technologies, the process of video quality prediction usually adopts the following method: first, the video to be predicted is divided into a series of video frames, and then these video frames are subjected to data enhancement processing, such as rotation, translation, flipping, etc. Then, the video frames after data enhancement are subjected to feature extraction and image content understanding, and finally the quality score of the video to be predicted is obtained. In other words, in the process of each video quality prediction for a video, it is necessary to infer each video frame in the video, and then perform image content understanding based on the inference results of all video frames, which greatly increases the computational burden in the video quality prediction process and reduces the efficiency of video quality prediction.

[0042] In view of this, an embodiment of the present application provides a video quality prediction method, which can know the clarity parameters, encoding parameters, current resolution parameters and image quality distortion indicators of the current video by obtaining the video information of the current video. The process does not need to extract video frames for the current video, nor does it need to process multiple video frames accordingly. Considering that the current video is obtained by encoding, transmitting and decoding the original video, that is, the current video is derived from the original video, therefore, in order to comprehensively and effectively estimate the image quality of the current video, the video information of the original video is obtained to determine the bit rate, original image quality score and original resolution parameters of the original video. Then, the trained video quality assessment model is called, and based on the video information of the current video and the video information of the original video, the video quality feature representation of the current video can be extracted, so as to accurately reflect the image quality stop loss of the current video relative to the original video, and obtain the current image quality score of the current video with high accuracy, and the current image quality score is used to characterize the image quality loss degree of the current video. In this way, in the process of predicting the image quality of the current video, there is no need to perform image understanding on each video frame in the current video, which reduces the computational burden and the amount of calculation, thereby improving the efficiency of video quality prediction.

[0043] The video quality prediction method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the video transmitter 102 communicates with the server 104 via a network. The video receiver 106 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other servers.

[0044] In some embodiments, in a long video scenario, if the original video is a live video, the video transmitter 102 encodes the original video to obtain video encoding data, and transmits the video encoding data and the video information of the original video to the server 104. The video information of the original video includes the bit rate, original image quality score and original resolution parameter of the original video. The server 104 transmits the received video encoding data and the video information of the original video to the video receiver 106, and the video receiver 106 decodes the received video encoding data to obtain the current video and determines the video information of the current video. The video information of the current video includes the clarity parameter, encoding parameter, current resolution parameter and image quality distortion index of the current video.

[0045] The video receiving end 106 calls the trained video quality assessment model, obtains the video quality feature representation of the current video according to the video information of the current video and the video information of the original video, and outputs the current picture quality score of the current video based on the video quality feature representation. The current picture quality score is used to characterize the degree of picture quality degradation of the current video.

[0046] In other embodiments, after the server 104 receives the video encoding data and the video information of the original video, in the short video scenario, the video encoding data can be directly decoded by the server 104, the decoded video can be directly used as the current video, and the video information of the current video can be determined. Then, the server 104 calls the trained video quality assessment model, obtains the video quality feature representation of the current video based on the video information of the current video and the video information of the original video, and outputs the current image quality score of the current video based on the video quality feature representation. The current image quality score is used to characterize the degree of image quality degradation of the current video. The server 104 returns the current image quality score and the current video to the video receiving end 106.

[0047] The video transmitter 102 and the video receiver 106 are different terminals. The terminals may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, and IoT devices. The IoT devices may be smart TVs, smart car devices, and projection devices. The server 104 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0048] In an exemplary embodiment, Figure 2 As shown, a video quality prediction method is provided, and the method is applied to a computer device (which can be Figure 1 The server 104 in the example (or the video receiving end 106) is used as an example to illustrate, including the following steps S202 to S206. Among them:

[0049] Step S202, obtaining video information of the current video. The current video originates from the original video. The current video is obtained by encoding, transmitting, and decoding the original video. The video information of the current video includes clarity parameters, encoding parameters, current resolution parameters, and image quality distortion indicators of the current video.

[0050] The current video refers to a video obtained by encoding, transmitting, and decoding the original video. The current video is a video to be subjected to video quality prediction, and can also be understood as a video obtained after decoding.

[0051] For example, in a live broadcast scenario, the video generated by the anchor account during the live broadcast is the original video. The original video is encoded by the anchor terminal logged in by the live broadcast account, and the obtained video encoding data is pushed to the server. The server pushes the video encoding data to the viewer client where the viewer account watching the live broadcast is located. At this time, the viewer client decodes the received video to obtain the current video.

[0052] The definition parameter of the current video is used to reflect the definition level of the current video. The current resolution parameter includes the height and width of the current video. The encoding parameter of the current video includes at least the encoding standard and the bit rate of the current video.

[0053] Coding standards refer to a series of specifications and technical requirements followed when compressing and encoding the current video, such as H.264, H.265, etc. These standards affect the compression efficiency and compatibility of the video.

[0054] The bit rate of the current video refers to the bit rate of the current video, reflecting the amount of data required to transmit or store the current video.

[0055] The image quality distortion index of the current video is used to measure the difference in image quality between the current video and the original video. The larger the value of the image quality distortion index, the lower the error and the better the quality of the current video. Exemplarily, the image quality distortion index can be determined based on the signal-to-noise ratio or the structural similarity index.

[0056] Exemplarily, after receiving a video quality prediction request for a current video, the computer device obtains video information of the current video from local storage.

[0057] For example, the video transmitter sends a video quality prediction request to the server, and the server determines the current video pointed to by the video quality prediction request and determines the video information of the current video.

[0058] For another example, the video receiving end obtains the video information of the current video from the local storage in response to a triggering operation of the video operation and maintenance personnel to perform video quality prediction on the current video.

[0059] Step S204, obtaining video information of the original video, where the video information of the original video includes the bit rate, original image quality score and original resolution parameters of the original video.

[0060] The bit rate of the original video refers to the bit rate of the original video, which reflects the amount of data required to transmit or store the original video.

[0061] The original video resolution parameters include the height and width of the original video.

[0062] The original image quality score of the original video is used to reflect the quality (image quality) of the original video. The higher the original image quality score, the better the quality of the original video.

[0063] Exemplarily, after receiving a video quality prediction request for a current video, the computer device determines the original video from which the current video originates, and obtains video information of the original video from local storage.

[0064] In some embodiments, obtaining video information of the current video and video information of the original video includes: obtaining a video quality prediction request, parsing the video quality prediction request, and determining the video identifier of the current video for video quality prediction; based on the video identifier, reading a corresponding video description file from local storage, and obtaining the video information of the current video and the video information of the original video from which the current video originates from the video description file.

[0065] The video quality prediction request is used to request a video quality prediction for the current video. The corresponding video description file refers to the video description file of the current video.

[0066] The video description file of the current video is used to store relevant video information corresponding to the current video, and the relevant video information at least includes the video information of the current video and the video information of the original video from which the current video originates. The video description file of the current video is used to predict the video quality of the current video.

[0067] For example, in response to a trigger operation of video quality prediction by a video operator for a current video, a video quality prediction request is generated. Alternatively, a video quality prediction request for the latest acquired current video is automatically generated at a preset time interval.

[0068] The computer device parses the obtained video quality prediction request to determine the video identifier of the current video to be predicted. The computer device queries the video description file that matches the video identifier from the pre-stored video description files corresponding to each decoded video. The computer device obtains the video information of the current video and the video information of the original video from which the current video originates from the queried video description file.

[0069] In this embodiment, the received video quality prediction request is parsed to determine the current video for which video quality prediction is to be performed, and based on the video identifier of the current video, the video information of the current video and the video information of the original video from which the current video originates are quickly queried from the local storage, thereby improving the efficiency of obtaining video information.

[0070] In some embodiments, Figure 3 FIG. 1 is a flow chart of steps for determining a video description file of a current video in an embodiment. The steps for generating a video description file of a current video include:

[0071] Step S302: receiving a video description file of an original video sent by a video transmitter and video encoding data obtained by encoding the original video, and parsing video information of the original video from the video description file of the original video.

[0072] Exemplarily, after acquiring the original video uploaded by the video creator, the video transmitter encodes the original video to obtain video encoding data. The video transmitter counts the bit rate, original image quality score and original resolution parameters of the original video to obtain the video information of the original video, and generates a video description file of the original video based on the video information and video encoding data of the original video. After receiving the video description file of the original video generated by the video transmitter, the computer device parses the video information of the original video from the video description file of the original video.

[0073] In some embodiments, the step of determining the original picture quality score includes: inputting multiple video frames in the original video into a picture quality assessment model based on video features for feature extraction, obtaining visual features of each video frame, and determining the original picture quality score of the original video based on the visual features of each video frame.

[0074] Among them, the video feature-based image quality assessment model is a video quality assessment model based on machine vision, which focuses on analyzing the visual content of video frames and assesses video quality by identifying and evaluating visual features such as texture, color, and contrast in video frames. Therefore, the video feature-based image quality assessment model can also be understood as a visual feature-based image quality assessment model.

[0075] The visual features of the video frame include at least texture features for reflecting texture details of the video frame, color features for reflecting color information of the video frame, and contrast features for reflecting contrast information of the video frame.

[0076] The original image quality score is used to measure the visual quality of the original video. The higher the original image quality score, the better the visual quality of the original video.

[0077] Exemplarily, the video transmitter performs frame processing on the original video and extracts multiple video frames at a preset frequency. The video transmitter calls the trained video feature-based image quality assessment model to extract the visual features of each video frame, and determines the original video quality score based on the visual features of each video frame.

[0078] It should be noted that this embodiment is implemented by the video sending end before the computer device obtains the current video. This embodiment is a preparatory step for predicting the video quality of the current video, that is, it is automatically completed by the video sending end after obtaining the original video, and the computer device does not need to perform image understanding on each video frame of the original video.

[0079] In this embodiment, visual features of video frames of the original video are extracted through a video feature-based image quality prediction model to obtain visual features of each video frame, thereby predicting the original image quality score of the original video in advance from the visual feature dimension.

[0080] Step S304, decoding the video encoding data to obtain the current video, and counting the clarity parameters, encoding parameters, current resolution parameters and image quality distortion indicators of the current video to obtain video information of the current video.

[0081] Exemplarily, the computer device determines the matching definition level based on the current network speed of the video receiving end, and obtains the video information of the current video based on the definition level, encoding standard, bit rate, current resolution parameter and image quality distortion index. For example, the lower the current network speed, the lower the definition level, that is, the less clear the video is.

[0082] In some embodiments, the step of determining the image quality distortion index includes: calculating the mean square error according to the pixel values ​​of the current video and the original video to obtain the image quality distortion index of the current video.

[0083] Exemplarily, a computer device obtains multiple original video frames in an original video, obtains a current video frame corresponding to the original video frame from the current video frame for each original video frame, and calculates a mean square error based on pixel values ​​of the original video frame and the current video frame to obtain a peak signal-to-noise ratio between the original video frame and the current video frame. The computer device calculates the mean of the peak signal-to-noise ratios corresponding to each original video frame to obtain an image quality distortion index of the current video.

[0084] For example, for any original video frame, after calculating the corresponding mean square error MSE, the peak signal-to-noise ratio PSNR is calculated based on the following formula:

[0085] PSNR = 10 × log10 ((MAX_I) 2 / MSE)

[0086] In the above formula, MAX_I is the maximum pixel value in the original video frame.

[0087] Among them, the peak signal-to-noise ratio is an indicator used to measure image quality. It can be understood that the larger the peak signal-to-noise ratio is, the more similar the video quality of the current video is to the original video, that is, the smaller the degree of image quality distortion of the current video is.

[0088] In other embodiments, the mean, variance and covariance between the original video frame and the current video frame are calculated according to their respective pixel values, and the structural similarity index corresponding to the original video frame is determined based on the mean, variance and covariance, and the mean of the structural similarity index corresponding to each original video frame is calculated to obtain the image quality distortion index of the current video.

[0089] Alternatively, the mean value of the structural similarity index and the mean value of the peak signal-to-noise ratio are fused to obtain the image quality distortion index of the current video.

[0090] In this embodiment, the mean square error is calculated based on the pixel values ​​of the current video and the original video to obtain the image quality distortion index of the current video, that is, the current video is compared with the original video from the distortion dimension, which accurately reflects the degree of image quality distortion of the current video.

[0091] Step S306: Generate a video description file of the current video based on the video information of the original video and the video information of the current video, and store the video description file of the current video and the video identifier of the current video in correspondence with each other.

[0092] In this embodiment, once the video encoding data and the video description file of the original video are obtained, the current video is obtained in advance based on the video encoding data to obtain the video information of the current video, so that the video description file of the current video can be generated in real time and stored. After the video identifier of the current video to be predicted for video quality is determined based on actual needs, the video information of the current video and the original video can be directly and quickly obtained from the local storage, which simplifies the entire video quality prediction process, saves the time spent on the entire process, and improves the efficiency of video quality prediction.

[0093] Step S206, calling the trained video quality assessment model, obtaining the video quality feature representation of the current video according to the video information of the current video and the video information of the original video, and outputting the current picture quality score of the current video based on the video quality feature representation. The current picture quality score is used to characterize the degree of picture quality degradation of the current video.

[0094] The trained video quality assessment model is used to predict the quality score of the current video. The trained video quality assessment model is a lightweight neural network model constructed by a fully connected network (FCN), that is, the model structure is simple, and it can quickly predict the quality of the current video. It is also easy to integrate into various video processing and transmission systems, providing users with real-time and efficient video quality prediction services.

[0095] Exemplarily, the trained video quality assessment model includes a plurality of sequentially connected feature extraction networks, each of which includes a corresponding fully connected layer.

[0096] The higher the current image quality score is, the lower the image quality degradation of the current video is; the lower the current image quality score is, the higher the image quality degradation of the current video is.

[0097] The video quality feature representation of the current video is used to represent the features of the video quality of the current video.

[0098] Optionally, the computer device calls a trained video quality assessment model, inputs the video information of the current video and the original video into the trained video quality assessment model, extracts the video quality feature representation of the current video through multiple feature extraction networks, and determines and outputs the current picture quality score of the current video based on the video quality feature representation, and determines the degree of picture quality degradation of the current video based on the current picture quality score.

[0099] Exemplarily, the computer device also obtains object behavior data about the current video, and inputs the object behavior data, the original video, and the video information of the current video into a trained video quality assessment model, and returns to extract the video quality feature representation step of the current video through multiple feature extraction networks to continue execution.

[0100] The above-mentioned object behavior data about the current video is obtained by counting the viewing data of users browsing the current video (such as viewing time, number of pauses, fast forward / rewind operations, etc.). The object behavior data can indirectly evaluate the picture quality and user satisfaction of the current video.

[0101] Based on this, in the process of determining the current picture quality score, the video information of the comprehensive video dimension (including the video information of the current video and the original video) and the data of the object behavior dimension related to the video can more comprehensively evaluate the video quality of the current video and further improve the accuracy of video quality prediction.

[0102] It should be noted that the network structure of the video quality assessment model based on video features in the previous article is completely different from that of the video quality assessment model in this step, and the inputs of each model are also different. Among them, the video quality assessment model based on video features predicts the video quality of the video to which the video frame belongs based on the visual features of the video frame. The video quality assessment model does not need to process the video features of the video frame, but only needs to obtain the video information of the current video and the original video, so the video quality prediction process is more efficient.

[0103] In some embodiments, a trained video quality assessment model is called to obtain a video quality feature representation of the current video based on the video information of the current video and the video information of the original video, including: inputting the video information of the current video and the video information of the original video into the first feature extraction network of a plurality of feature extraction networks connected in sequence to perform feature extraction to obtain the first intermediate feature; starting from the second feature extraction network, through the current feature extraction network, feature extraction is performed on the intermediate feature output by the previous feature extraction network to obtain the corresponding intermediate feature, until the last feature extraction network outputs the video quality feature representation of the current video.

[0104] Among them, Figure 4 FIG. 1 is a schematic diagram of a model structure of a video quality assessment model in an embodiment. The video quality assessment model includes an input layer, a feature extraction network 1, a feature extraction network 2, ..., a feature extraction network M and an output layer connected in sequence. Where M is a positive integer.

[0105] Exemplarily, each feature extraction network includes corresponding fully connected layers, normalization layers, and activation layers connected in sequence. Among them, the fully connected layer is used to map the feature representation input to the fully connected layer to the corresponding feature space, and the corresponding feature representation is output. Then, a normalization layer, such as BatchNorm1d (batch normalization layer), is used to normalize the output of each feature dimension. Finally, an activation layer, such as a ReLU activation function, is used to introduce nonlinearity, so that the network can approximate any nonlinear function, thereby enhancing the representation ability of the network.

[0106] For example, Figure 5 FIG. 4 is a schematic diagram of a model structure of a video quality assessment model in another embodiment. Figure 5There are three feature extraction networks, and the dimension of the output of the fully connected layer in each feature extraction network is different. Fully connected layer 1 realizes mapping the feature representation input to fully connected layer 1 to 128-dimensional space, that is, the output dimension is 128; fully connected layer 2 realizes mapping the feature representation input to fully connected layer 2 to 64-dimensional space, that is, the output dimension is 64; fully connected layer 3 realizes mapping the feature representation input to fully connected layer 3 to 32-dimensional space, that is, the output dimension is 32. The batch normalization layer in each feature extraction network can be the same or different, and the activation layer in each feature extraction network can be the same or different.

[0107] A dropout layer is also introduced between feature extraction network 1 and feature extraction network 2. The dropout layer is a regularization process used to prevent overfitting of the neural network. During the training process, the dropout layer randomly sets the output of some neurons to zero, which can reduce the complex co-adaptation relationship between neurons. In the figure, 0.5 means that there is a 50% probability of discarding the output of a certain neuron, that is, half of the neurons will not be activated in each forward propagation.

[0108] Exemplarily, the computer device maps the adaptability information of the current video and the original video to a 128-dimensional space through a feature extraction network 1, passes through a batch normalization layer and an activation layer in sequence, and outputs a 128-dimensional intermediate feature 1.

[0109] After the 128-dimensional intermediate feature 1 is input into the discard layer, the discarded intermediate feature is obtained. The discarded intermediate feature is mapped to a 64-dimensional space through the feature extraction network 2, and then passes through the batch normalization layer and the activation layer in sequence to output the 64-dimensional intermediate feature 2.

[0110] The intermediate feature 2 is mapped to a 32-dimensional space through the feature extraction network 3, and passes through a batch normalization layer and an activation layer in sequence to output a 32-dimensional intermediate feature 3, which is the video quality feature representation of the current video.

[0111] The steps for determining the number of feature extraction networks in the video quality assessment model include:

[0112] 1. Pre-build multiple candidate video quality assessment models, and the number of feature extraction networks in each candidate assessment model is different.

[0113] 2. For each candidate video quality assessment model, obtain the corresponding training sample, the training sample includes the video information of the target sample video and the video information of the original sample video. The target sample video is derived from the original sample video, and the target sample video is obtained by encoding, transmitting, and decoding the original sample video.

[0114] The candidate video quality assessment model is trained based on the training samples and the corresponding labeled data to obtain a trained candidate video quality assessment model.

[0115] 3. For each trained candidate video quality assessment model, obtain the corresponding test sample, which includes the video information of the target test video and the video information of the original test video. The target test video is derived from the original test video, and the target test video is obtained by encoding, transmitting, and decoding the original test video.

[0116] 4. Input the test sample into the corresponding trained candidate video quality assessment model to obtain the predicted image quality score. Based on the predicted image quality score and the labeled data of the test sample, the prediction accuracy of the trained candidate video quality assessment model is calculated.

[0117] 5. The number of feature extraction networks in the candidate video quality assessment model corresponding to the highest prediction accuracy is the number of feature extraction networks in the finally trained video quality assessment model, that is, the candidate video quality assessment model corresponding to the highest prediction accuracy is the trained video quality assessment model.

[0118] The training model of each candidate video quality assessment model mentioned above can refer to the training process of the video quality assessment model below.

[0119] In this embodiment, the video information of the current video and the video information of the original video are input into the first feature extraction network for feature extraction to obtain the first intermediate feature. In this way, the intermediate feature output by the previous feature extraction network is subjected to feature extraction to obtain the corresponding intermediate feature. In this way, the video quality feature representation of the current video can be quickly and accurately obtained through multiple feature extractions. In addition, a plurality of sequentially connected feature extraction networks construct a lightweight video quality assessment model, so that the model can respond quickly and process video quality assessment tasks in real time. This design is not only suitable for real-time monitoring of online video streams, but also easy to integrate into various video playback and transmission devices.

[0120] In some embodiments, Figure 6 FIG. 1 is a flow chart of a model training step in an embodiment. The video quality assessment model training step includes:

[0121] Step S602, obtaining a training sample, the training sample includes video information of a target sample video and video information of an original sample video, the target sample video is derived from the original sample video, the target sample video is obtained by encoding, transmitting, and decoding the original sample video, the video information of the target sample video includes clarity parameters, encoding parameters, resolution parameters, and image quality distortion indicators of the target sample video, and the video information of the original sample video includes the bit rate, original image quality score, and resolution parameters of the original sample video.

[0122] Step S604, according to the video information of the target sample video and the video information of the original sample video, a video quality feature representation of the target sample video is obtained through a video quality assessment model, and a picture quality score of the target sample video is output based on the video quality feature representation.

[0123] Among them, the video quality assessment model is a neural network model to be trained.

[0124] Exemplarily, the video quality assessment model may be any of the candidate video assessment models mentioned above.

[0125] Exemplarily, after the number of feature extraction networks is determined based on actual needs, a video quality assessment model is constructed based on the number.

[0126] Step S606: Based on the difference between the labeled data of the training sample and the image quality score of the target sample video, the video quality assessment model is trained to obtain a trained video quality assessment model.

[0127] Among them, the labeled data of the training samples refers to the data that annotates the degree of image quality degradation of the target sample video. For example, the degree of image quality degradation includes 5 levels, from low to high, namely level 1 (corresponding to the image quality score range of 100-80), level 2 (corresponding to the image quality score range of 79-60), level 3 (corresponding to the image quality score range of 59-40), level 4 (corresponding to the image quality score range of 39-20), and level 5 (corresponding to the image quality score range of 10-0). The higher the level, the worse the corresponding video quality and the lower the image quality score.

[0128] For example, for training sample 1, the training sample is labeled as level 1, so the labeled data of level 1 is set to 1, and the labeled data of levels 2 to 5 are all set to 0.

[0129] Exemplarily, sample amplification is performed based on the training sample to determine the corresponding amplified sample, the amplified sample includes video information of the amplified target sample video and the amplified original sample video, and the amplified original sample video is the same as the original sample video in the training sample. The amplified target sample video and the target sample video each have different corresponding video information, and the annotation data corresponding to the amplified target sample video (i.e., the annotation data of the amplified sample) is different from the annotation data of the training sample.

[0130] The amplified sample is input into the video quality assessment model to obtain the image quality score of the amplified target sample video, the difference between the labeled data of the training sample and the image quality score of the target sample video, and the difference between the labeled data of the amplified sample and the image quality score of the amplified target sample video are superimposed, and a target loss is constructed to minimize the target loss. The video quality assessment model is trained to obtain a trained video quality assessment model.

[0131] In some embodiments, the step of determining the labeled data of the training sample includes: inputting multiple video frames in the target sample video into a video feature-based image quality assessment model for feature extraction to obtain visual features of each video frame, determining the image quality score of the target sample video based on the visual features of each video frame, and determining the labeled data of the training sample based on the image quality score of the target sample video.

[0132] Exemplarily, during the training phase, the computer device performs frame processing on the target sample video and extracts multiple video frames at a preset frequency. The computer device calls a video feature-based image quality assessment model to extract the visual features of each video frame, and determines the image quality score of the target sample video based on the visual features of each video frame. The computer device determines a matching grade based on the image quality score interval to which the image quality score belongs, and determines the labeled data for the training sample based on the grade.

[0133] In other embodiments, the computer device may also calculate the peak signal-to-noise ratio between the target sample video and the target original video, or calculate the structural similarity index between the target sample video and the target original video to obtain the corresponding calculation result. Based on the calculation result and the image quality score of the target sample video calculated above, the target image quality score of the target sample video is determined, and based on the image quality score interval to which the target image quality score belongs, the matching grade is determined, and based on the grade, the labeled data of the training sample is determined.

[0134] Exemplarily, the computer device determines the cross entropy loss based on the difference between the labeled data of the training sample and the image quality score of the target sample video, and performs multiple iterations based on the cross entropy loss until the last iteration is completed to obtain a trained video quality assessment model.

[0135] Alternatively, based on the cross entropy loss, the model parameters are adjusted according to the corresponding learning rate, and the next iteration is performed based on the adjusted parameters until the parameter adjustment is stopped when the cross entropy loss is minimized, thereby obtaining a trained video quality assessment model.

[0136] Among them, a larger learning rate can be used to accelerate convergence in the early stage of training (starting the iteration process), and the learning rate can be gradually reduced in the later stage of training (the reduction of cross entropy loss becomes smaller) to stabilize the model. In this way, based on the learning rate decay strategy and the early stopping mechanism (before the preset number of iterations is reached, the cross entropy loss is minimized, that is, the cross entropy loss no longer changes, then the model training is stopped), it helps the model achieve better prediction results.

[0137] In this way, during the model training stage, the image quality score of the target sample video is predicted in advance through the image quality assessment model based on video features, so that the target sample video can be accurately labeled, which can ensure the accuracy of subsequent video quality assessment model training.

[0138] In this embodiment, the video information of the target sample video and the original sample video in the training sample is input into the video quality assessment model to learn the deep feature representation of the video quality of the target sample video, that is, to obtain the video quality feature representation, thereby determining the image quality score of the target sample video. According to the difference between the labeled data of the training sample and the image quality score of the target sample video, it is possible to verify in real time whether the model training is completed, and to accurately and effectively train the video quality assessment model, thereby improving the accuracy and effectiveness of the model process.

[0139] In the above-mentioned video quality prediction method, by obtaining the video information of the current video, the clarity parameters, encoding parameters, current resolution parameters and image quality distortion index of the current video can be known. This process does not require video frame extraction for the current video, nor does it require corresponding processing of multiple video frames. Considering that the current video is obtained by encoding, transmitting and decoding the original video, that is, the current video is derived from the original video, therefore, in order to comprehensively and effectively estimate the image quality of the current video, the video information of the original video is obtained to determine the bit rate, original image quality score and original resolution parameters of the original video. Then, the trained video quality assessment model is called, and based on the video information of the current video and the video information of the original video, the video quality feature representation of the current video can be extracted, so as to accurately reflect the image quality stop loss of the current video relative to the original video, and obtain the current image quality score of the current video with high accuracy. The current image quality score is used to characterize the image quality loss degree of the current video. In this way, in the process of predicting the image quality of the current video, it is not necessary to perform image understanding on each video frame in the current video, which reduces the computational burden and the amount of computation, and also improves the efficiency of video quality prediction.

[0140] In some embodiments, Figure 7 FIG. 1 is a flow chart of determining a target image quality score in one embodiment, wherein the method further includes:

[0141] Step S702, obtaining transmission network parameters corresponding to the original video and object behavior data about the current video.

[0142] Among them, the transmission network parameters are used to evaluate the transmission network between the video sender and the server, and the transmission network between the server and the video receiver. The transmission network parameters at least include network delay, network speed, and freeze frequency.

[0143] Step S704: predict the video quality of the current video based on the encoding parameters, transmission network parameters and object behavior data of the current video to obtain a reference image quality score of the current video.

[0144] Exemplarily, a computer device performs data normalization processing on the encoding parameters, transmission network parameters and object behavior data of the current video, respectively, and then splices them to obtain spliced ​​data, and calls a picture quality assessment model based on video-related information to predict the video quality of the current video based on the spliced ​​data to obtain a reference picture quality score for the current video.

[0145] It should be noted that transmission network parameters and object behavior data belong to video-related information, not video information. The focus is on the impact of compression, transmission, encoding and decoding processes on video quality.

[0146] The image quality assessment model based on video-related information, the image quality assessment model based on video features mentioned above, and the trained video quality assessment model have different model structures, and their respective video quality prediction processes are also different.

[0147] The data normalization processing mentioned above may be to unify the values ​​of the encoding parameters of the current video, the values ​​of the transmission network parameters, and the values ​​in the object behavior data into a preset range, for example, 0 to 1.

[0148] In some embodiments, based on the encoding parameters, transmission network parameters and object behavior data of the current video, the video quality of the current video is predicted to obtain a reference image quality score of the current video, including: calling an image quality assessment model based on video-related information to respectively determine the image quality scores corresponding to the encoding parameters, transmission network parameters and object behavior data of the current video; fusing the image quality scores corresponding to the encoding parameters, transmission network parameters and object behavior data of the current video to obtain a reference image quality score of the current video.

[0149] In other embodiments, it may also be: calling a picture quality assessment model based on video-related information, performing feature extraction on the encoding parameters, transmission network parameters and object behavior data of the current video respectively, obtaining a first feature representation reflecting the encoding parameters, a second feature representation reflecting the transmission network parameters and a third feature representation reflecting the object behavior data, fusing the first feature representation, the second feature representation and the third feature representation to obtain a fused feature representation, determining a reference picture quality score of the current video based on the fused feature representation, and outputting the reference picture quality score.

[0150] In this embodiment, the image quality evaluation model based on video-related information determines the image quality scores of the encoding dimension, transmission network dimension and object behavior dimension respectively through the encoding parameters, transmission network parameters and object behavior data of the current video. In this way, the image quality scores of multiple dimensions can be integrated to obtain a reference image quality score, thereby accurately reflecting the impact of compression, transmission, encoding and decoding processes on video quality.

[0151] Step S706, the reference image quality score and the current image quality score of the current video are integrated to obtain a target image quality score of the current video, where the target image quality score is used to characterize the degree of image quality degradation of the current video.

[0152] Exemplarily, a computer device determines the weights corresponding to the reference image quality score and the current image quality score of the current video, and weights the reference image quality score and the current image quality score of the current video according to their respective weights to obtain a target image quality score for the current video. The target image quality score is used to characterize the degree of image quality degradation of the current video.

[0153] Exemplarily, the weight of the reference image quality score and the weight of the current image quality score may be equal or different, such as the weight of the current image quality score is greater than the weight of the reference image quality score.

[0154] like Figure 8 FIG. 1 is a schematic diagram of video quality prediction in an embodiment. The process of steps S702 to S706 can be referred to Figure 8 The process of video quality prediction.

[0155] In this embodiment, based on the encoding parameters of the current video, the transmission network parameters and the object behavior data, the encoding dimension, the transmission network dimension and the object behavior dimension are integrated to predict the video quality of the current video, and obtain the reference quality score of the current video to reflect the impact of compression, transmission, encoding and decoding processes on the video quality. Then, by integrating the reference quality score of the current video and the current quality score, the degree of quality loss of the current video relative to the original video can be more comprehensively and accurately evaluated, and a more accurate target quality score can be obtained, thereby improving the accuracy of video quality prediction.

[0156] The present application also provides an application scenario, which applies the above-mentioned video quality prediction method. Specifically, the application of the video quality prediction method in the application scenario is as follows: in a video push scenario, such as a streaming media server platform, any browsing client where a browsing user is located can use the video quality prediction method provided in the embodiment of the present application to efficiently and accurately predict the current quality score of the current video. Specifically, the browsing client obtains the video currently being browsed by the browsing user, which is the current video, and obtains video information of the current video. The current video is derived from the original video. The current video is obtained by encoding, transmitting, and then decoding the original video. The video information of the current video includes the clarity parameters, encoding parameters, current resolution parameters, and image quality distortion indicators of the current video; the video information of the original video is obtained, and the video information of the original video includes the bit rate, original image quality score, and original resolution parameters of the original video; the trained video quality assessment model is called, and the video quality feature representation of the current video is obtained according to the video information of the current video and the video information of the original video. Based on the video quality feature representation, the current image quality score of the current video is output, and the current image quality score is used to characterize the degree of image quality degradation of the current video. If the current image quality score is lower than the preset score, feedback is given to the streaming media server platform.

[0157] Of course, it is not limited to this. The video quality prediction method provided in this application can also be used by video content providers to predict the video quality of released videos, and can also be used in video compression and transmission. It can also be used in online education platforms, and can also be used in scenarios where the video quality of generated videos is monitored to analyze the video stream in real time, promptly detect the degradation of image quality, and promptly respond to and optimize the video quality.

[0158] The above application scenarios are merely illustrative. It is to be understood that the video quality prediction methods provided in the embodiments of the present application are not limited to the above scenarios.

[0159] In a specific embodiment, Fig. 9 FIG. 1 is a schematic diagram of a video quality prediction system architecture in an embodiment. The video quality prediction system involves a video transmitter and a computer device, which can be a server or a video receiver. The specific steps include:

[0160] For the training phase, refer to steps 9.1 to 9.3 below:

[0161] Step 9.1: The computer device obtains training samples.

[0162] Among them, the training sample includes video information of the target sample video and video information of the original sample video. The target sample video is derived from the original sample video. The target sample video is obtained by encoding, transmitting and decoding the original sample video. The video information of the target sample video includes clarity parameters of the target sample video, encoding parameters of the target sample video, resolution parameters of the target sample video and image quality distortion indicators of the target sample video. The video information of the original sample video includes the bit rate of the original sample video, the original image quality score of the original sample video and the resolution parameters of the original sample video.

[0163] Step 9.2: The computer device determines the labeled data of the training samples.

[0164] Optionally, the computer device calls a trained image quality assessment model based on video features, and inputs multiple video frames in the target sample video into the image quality assessment model based on video features for feature extraction to obtain visual features of each video frame, determines the image quality score of the target sample video based on the visual features of each video frame, and determines the annotation data of the training sample based on the image quality score of the target sample video.

[0165] Step 9.3: The computer device trains the video quality assessment model based on the training samples and the labeled data to obtain a trained video quality assessment model.

[0166] Optionally, the computer device obtains a video quality feature representation of the target sample video through a video quality assessment model based on the video information of the target sample video and the video information of the original sample video, and outputs a picture quality score for the target sample video based on the video quality feature representation; and trains the video quality assessment model based on the difference between the labeled data of the training sample and the picture quality score of the target sample video to obtain a trained video quality assessment model.

[0167] Among them, the trained video quality assessment model includes multiple feature extraction networks connected in sequence.

[0168] For the inference phase, refer to steps 9.4 to 9.9 below:

[0169] Step 9.4: The computer device obtains the video description file of the original video sent by any video sending end.

[0170] The steps of generating the video description file of the original video include:

[0171] The video transmitter calls the trained image quality assessment model based on video features to determine the original image quality score. The video transmitter counts the bit rate of the original video and the original resolution parameters of the original video, and generates the video information of the original video based on the original image quality score, the bit rate of the original video and the original resolution parameters.

[0172] The video transmitter encodes the original video to obtain video encoding data, and generates a video description file of the original video based on the video information and the video encoding data of the original video. The video transmitter sends the video description file of the original video to a computer device for storage.

[0173] Among them, the specific steps for generating the original picture quality score include: the video sending end inputs multiple video frames in the original video into a picture quality assessment model based on video features for feature extraction, obtains the visual features of each video frame, and determines the original picture quality score of the original video based on the visual features of each video frame.

[0174] Step 9.5: The computer device decodes the video encoding data to obtain the current video, generates a video description file of the current video, and stores the identifier of the current video and the generated video description file in correspondence.

[0175] After obtaining the video description file of the original video sent by each video sending end, the computer device generates the corresponding current video. For each current video i, the computer device generates the corresponding video description file of the current video. The specific generation process is as follows:

[0176] Step 9.5.1: For each received video description file of the original video, obtain video encoding data and video information of the original video from the video description file of the original video.

[0177] Step 9.5.2: Decode the video encoding data to obtain the current video.

[0178] Step 9.5.3: Count the video information of the current video, and generate a video description file of the current video based on the respective video information of the original video and the current video.

[0179] The video information of the current video includes a definition parameter of the current video, an encoding parameter of the current video, a current resolution parameter of the current video, and an image quality distortion index of the current video.

[0180] The step of determining the image quality distortion index includes: the computer device calculates the mean square error according to the pixel values ​​of the current video and the original video to obtain the image quality distortion index of the current video.

[0181] Step 9.6: The computer device obtains a video quality prediction request.

[0182] Step 9.7: The computer device obtains the video description file of the current video from the local storage.

[0183] Optionally, the computer device parses the video quality prediction request to determine the video identifier of the current video for video quality prediction. Based on the video identifier, the computer device reads the corresponding video description file from the local storage, and obtains the video information of the current video and the video information of the original video from which the current video is derived from the video description file.

[0184] Step 9.8: The computer device obtains the video information of the current video and the original video from the acquired video description file.

[0185] Step 9.9: Call the trained video quality assessment model to determine the current image quality score based on the video information of the current video and the original video.

[0186] Optionally, the computer device inputs the video information of the current video and the video information of the original video into the first feature extraction network of multiple sequentially connected feature extraction networks to perform feature extraction to obtain the first intermediate feature; starting from the second feature extraction network, the intermediate feature output by the previous feature extraction network is extracted through the current feature extraction network to obtain the corresponding intermediate feature, until the last feature extraction network outputs the video quality feature representation of the current video; based on the video quality feature representation, a current picture quality score for the current video is output, and the current picture quality score is used to characterize the degree of picture quality degradation of the current video.

[0187] Of course, it can also be: the computer device obtains the transmission network parameters corresponding to the original video, and the object behavior data about the current video; calls the image quality assessment model based on the video-related information to determine the image quality scores corresponding to the encoding parameters, transmission network parameters and object behavior data of the current video respectively; integrates the encoding parameters, transmission network parameters and image quality scores corresponding to the object behavior data of the current video to obtain a reference image quality score for the current video; integrates the reference image quality score and the current image quality score of the current video to obtain a target image quality score for the current video, and the target image quality score is used to characterize the degree of image quality degradation of the current video.

[0188] In this embodiment, by obtaining the video information of the current video, the clarity parameters, encoding parameters, current resolution parameters and image quality distortion index of the current video can be known. This process does not require video frame extraction for the current video, nor does it require corresponding processing of multiple video frames. Considering that the current video is obtained by encoding, transmitting and decoding the original video, that is, the current video is derived from the original video, therefore, in order to make a comprehensive and effective estimate of the image quality of the current video, the video information of the original video is obtained to determine the bit rate, original image quality score and original resolution parameters of the original video. Then, the trained video quality assessment model is directly called, and based on the video information of the current video and the video information of the original video, the video quality feature representation of the current video can be learned, so as to accurately reflect the image quality stop loss of the current video relative to the original video, and obtain a current image quality score of the current video with high accuracy. The current image quality score is used to characterize the degree of image quality loss of the current video. In this way, in the process of predicting the picture quality of the current video, there is no need to perform image understanding on each video frame in the current video, which reduces the computational burden and the amount of calculation. As a result, the video quality prediction process can be completed in a very short time, thereby improving the efficiency of video quality prediction.

[0189] In addition, it can provide users with instant quality feedback and help content providers and service operators optimize video encoding and transmission strategies, thereby providing a higher quality video viewing experience to improve user satisfaction. In addition, this application also simplifies the input amount of the video quality prediction model and greatly reduces the complexity of data processing. This method not only reduces the demand for storage and computing resources, but also improves the scalability and applicability of the system.

[0190] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0191] Based on the same inventive concept, the embodiment of the present application also provides a video quality prediction device for implementing the above-mentioned video quality prediction method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more video quality prediction device embodiments provided below can refer to the above-mentioned limitations on the video quality prediction method, and will not be repeated here.

[0192] In an exemplary embodiment, Fig.10 As shown, a video quality prediction device 1000 is provided, comprising: a video information acquisition module 1002 and a quality score acquisition module 1004, wherein:

[0193] The video information acquisition module 1002 is used to acquire video information of the current video. The current video is derived from the original video. The current video is obtained by encoding, transmitting, and decoding the original video. The video information of the current video includes the definition parameters, encoding parameters, current resolution parameters, and image quality distortion indicators of the current video.

[0194] The video information acquisition module 1002 is further used to acquire video information of the original video, where the video information of the original video includes the bit rate, original image quality score and original resolution parameters of the original video;

[0195] The image quality score acquisition module 1004 is used to call the trained video quality assessment model, obtain the video quality feature representation of the current video based on the video information of the current video and the video information of the original video, and output the current image quality score of the current video based on the video quality feature representation. The current image quality score is used to characterize the degree of image quality degradation of the current video.

[0196] In some embodiments, the video information acquisition module 1002 is also used to obtain a video quality prediction request, and parse the video quality prediction request to determine the video identifier of the current video for video quality prediction; based on the video identifier, read the corresponding video description file from the local storage, and obtain the video information of the current video and the video information of the original video from which the current video originates from the video description file.

[0197] In some embodiments, the video information acquisition module 1002 is also used to receive a video description file of the original video sent by the video sending end and video encoding data obtained by encoding the original video, and parse the video information of the original video from the video description file of the original video; decode the video encoding data to obtain the current video, and count the clarity parameters, encoding parameters, current resolution parameters and image quality distortion indicators of the current video to obtain the video information of the current video; generate a video description file of the current video based on the video information of the original video and the video information of the current video, and store the video description file of the current video in correspondence with the video identifier of the current video.

[0198] In some embodiments, the video information acquisition module 1002 is further used to calculate the mean square error according to the pixel values ​​of the current video and the original video to obtain the image quality distortion index of the current video.

[0199] In some embodiments, the video information acquisition module 1002 is also used to input multiple video frames in the original video into a video feature-based image quality assessment model for feature extraction, obtain visual features of each video frame, and determine the original image quality score of the original video based on the visual features of each video frame.

[0200] In some embodiments, the trained video quality assessment model includes multiple feature extraction networks connected in sequence, and the image quality score acquisition module 1004 is used to input the video information of the current video and the video information of the original video into the first feature extraction network among the multiple feature extraction networks connected in sequence for feature extraction to obtain the first intermediate feature; starting from the second feature extraction network, the intermediate feature output by the previous feature extraction network is extracted through the current feature extraction network to obtain the corresponding intermediate feature, until the last feature extraction network outputs the video quality feature representation of the current video.

[0201] In some embodiments, the device also includes a picture quality score fusion module, which is used to obtain the transmission network parameters corresponding to the original video and the object behavior data about the current video; based on the encoding parameters, transmission network parameters and object behavior data of the current video, the video quality of the current video is predicted to obtain a reference picture quality score of the current video; the reference picture quality score of the current video and the current picture quality score are integrated to obtain a target picture quality score of the current video, and the target picture quality score is used to characterize the degree of picture quality degradation of the current video.

[0202] In some embodiments, the image quality score fusion module is also used to call an image quality assessment model based on video-related information to respectively determine the image quality scores corresponding to the encoding parameters, transmission network parameters and object behavior data of the current video; and fuse the image quality scores corresponding to the encoding parameters, transmission network parameters and object behavior data of the current video to obtain a reference image quality score for the current video.

[0203] In some embodiments, the device also includes a model training module, which is used to obtain training samples, wherein the training samples include video information of a target sample video and video information of an original sample video. The target sample video is derived from the original sample video, and the target sample video is obtained by encoding, transmitting, and decoding the original sample video. The video information of the target sample video includes clarity parameters, encoding parameters, resolution parameters, and image quality distortion indicators of the target sample video, and the video information of the original sample video includes the bit rate, original image quality score, and resolution parameters of the original sample video. According to the video information of the target sample video and the video information of the original sample video, a video quality feature representation of the target sample video is obtained through a video quality assessment model, and a picture quality score of the target sample video is output based on the video quality feature representation. Based on the difference between the labeled data of the training sample and the picture quality score of the target sample video, the video quality assessment model is trained to obtain a trained video quality assessment model.

[0204] In some embodiments, a model training module is used to input multiple video frames in a target sample video into a video feature-based image quality assessment model for feature extraction to obtain visual features of each video frame, determine the image quality score of the target sample video based on the visual features of each video frame, and determine the annotation data of the training sample based on the image quality score of the target sample video.

[0205] Each module in the above-mentioned video quality prediction device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0206] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.11As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video quality prediction method is implemented.

[0207] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0208] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0209] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0210] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0211] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0212] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0213] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0214] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video quality prediction method, characterized in that: The method comprises: Acquire video information of a current video, where the current video is derived from an original video, and the current video is obtained by encoding, transmitting, and decoding the original video, and the video information of the current video includes a definition parameter, an encoding parameter, a current resolution parameter, and a picture quality distortion index of the current video; Acquire video information of the original video, where the video information of the original video includes a bit rate, an original picture quality score, and an original resolution parameter of the original video; Calling a trained neural network-based video quality assessment model, obtaining a video quality feature representation of the current video according to the video information of the current video and the video information of the original video, and outputting a current image quality score of the current video based on the video quality feature representation, wherein the current image quality score is used to characterize the degree of image quality degradation of the current video. The video quality assessment model does not need to process the video features of the video frame, but only needs to obtain the video information of the current video and the original video respectively; The video quality assessment model training step includes: Acquire a training sample, wherein the training sample includes video information of a target sample video and video information of an original sample video, wherein the target sample video is obtained by encoding, transmitting, and decoding the original sample video, wherein the video information of the target sample video includes a clarity parameter, an encoding parameter, a resolution parameter, and an image quality distortion index of the target sample video, and the video information of the original sample video includes a bit rate, an original image quality score, and a resolution parameter of the original sample video; Obtaining a video quality feature representation of the target sample video according to the video information of the target sample video and the video information of the original sample video through a video quality assessment model, and outputting a picture quality score of the target sample video based on the video quality feature representation; Based on the difference between the labeled data of the training sample and the image quality score of the target sample video, updating the model parameters to obtain a trained video quality assessment model; Among them, the step of determining the labeled data of the training sample includes: inputting multiple video frames in the target sample video into a picture quality assessment model based on video features for feature extraction to obtain visual features of each video frame, determining the picture quality score of the target sample video according to the visual features of each video frame, and determining the labeled data of the training sample based on the picture quality score of the target sample video.

2. The method according to claim 1, characterized in that Get the video information of the current video and the original video, including: Obtaining a video quality prediction request, parsing the video quality prediction request, and determining a video identifier of a current video for which video quality prediction is to be performed; Based on the video identifier, a corresponding video description file is read from local storage, and video information of the current video and video information of an original video from which the current video originates are obtained from the video description file.

3. The method according to claim 2, characterized in that Before obtaining the video information of the current video, the steps of generating a video description file of the current video include: Receiving a video description file of an original video sent by a video transmitter and video encoding data obtained by encoding the original video, and parsing video information of the original video from the video description file of the original video; Decoding the video encoding data to obtain a current video, and collecting statistics on a definition parameter, encoding parameter, current resolution parameter, and image quality distortion index of the current video to obtain video information of the current video; Based on the video information of the original video and the video information of the current video, a video description file of the current video is generated, and the video description file of the current video and the video identifier of the current video are stored in correspondence.

4. The method according to claim 1, characterized in that: The step of determining the image quality distortion index comprises: According to the pixel values ​​of the current video and the original video, a mean square error is calculated to obtain an image quality distortion index of the current video.

5. The method according to claim 1, characterized in that: The step of determining the original image quality score comprises: Multiple video frames in the original video are input into a picture quality assessment model based on video features for feature extraction to obtain visual features of each video frame, and the original picture quality score of the original video is determined according to the visual features of each video frame.

6. The method according to claim 1, characterized in that The trained neural network-based video quality assessment model includes a plurality of sequentially connected feature extraction networks, and calling the trained neural network-based video quality assessment model to obtain a video quality feature representation of the current video according to the video information of the current video and the video information of the original video includes: Inputting the video information of the current video and the video information of the original video into a first feature extraction network among the plurality of sequentially connected feature extraction networks to perform feature extraction, thereby obtaining a first intermediate feature; Starting from the second feature extraction network, the intermediate features output by the previous feature extraction network are extracted through the current feature extraction network to obtain the corresponding intermediate features, until the last feature extraction network outputs the video quality feature representation of the current video.

7. The method according to claim 1, characterized in that The method further comprises: Acquire transmission network parameters corresponding to the original video and object behavior data about the current video; Predicting the video quality of the current video based on the encoding parameters of the current video, the transmission network parameters, and the object behavior data to obtain a reference picture quality score of the current video; The reference image quality score of the current video and the current image quality score are integrated to obtain a target image quality score of the current video, where the target image quality score is used to characterize the degree of image quality degradation of the current video.

8. The method according to claim 7, characterized in that The predicting the video quality of the current video based on the encoding parameter of the current video, the transmission network parameter and the object behavior data to obtain a reference picture quality score of the current video includes: Calling a picture quality assessment model based on video-related information to respectively determine picture quality scores corresponding to the encoding parameters of the current video, the transmission network parameters, and the object behavior data; The encoding parameters of the current video, the transmission network parameters, and the image quality scores corresponding to the object behavior data are integrated to obtain a reference image quality score of the current video.

9. A video quality prediction device, characterized in that: The device comprises: A video information acquisition module is used to acquire video information of a current video, wherein the current video is derived from an original video, and the current video is obtained by encoding, transmitting, and decoding the original video, and the video information of the current video includes a definition parameter, encoding parameter, current resolution parameter, and image quality distortion index of the current video; The video information acquisition module is further used to acquire video information of the original video, where the video information of the original video includes a bit rate, an original picture quality score, and an original resolution parameter of the original video; A picture quality score acquisition module is used to call a trained neural network-based video quality assessment model, obtain a video quality feature representation of the current video according to the video information of the current video and the video information of the original video, and output a current picture quality score of the current video based on the video quality feature representation, wherein the current picture quality score is used to characterize the degree of picture quality degradation of the current video. The video quality assessment model does not need to process the video features of the video frame, but only needs to obtain the video information of the current video and the original video respectively; The device further comprises a model training module, which is used to obtain training samples, wherein the training samples comprise video information of a target sample video and video information of an original sample video, wherein the target sample video is obtained by encoding, transmitting and decoding the original sample video, wherein the video information of the target sample video comprises a clarity parameter, encoding parameter, resolution parameter and image quality distortion index of the target sample video, and the video information of the original sample video comprises a bit rate, an original image quality score and a resolution parameter of the original sample video; a video quality feature representation of the target sample video is obtained according to the video information of the target sample video and the video information of the original sample video through a video quality assessment model, and an image quality score of the target sample video is output based on the video quality feature representation; and a model parameter is updated based on the difference between the labeled data of the training sample and the image quality score of the target sample video to obtain a trained video quality assessment model; Among them, the model training module is used to input multiple video frames in the target sample video into a picture quality assessment model based on video features for feature extraction, obtain the visual features of each video frame, determine the picture quality score of the target sample video according to the visual features of each video frame, and determine the annotation data of the training sample based on the picture quality score of the target sample video.

10. The device according to claim 9, characterized in that The video information acquisition module is further used to obtain a video quality prediction request, and parse the video quality prediction request to determine a video identifier of a current video for which video quality prediction is to be performed; Based on the video identifier, a corresponding video description file is read from local storage, and video information of the current video and video information of an original video from which the current video originates are obtained from the video description file.

11. The device according to claim 10, characterized in that Before acquiring the video information of the current video, the video information acquisition module is further used to receive a video description file of the original video sent by the video sending end and video encoding data obtained by encoding the original video, and parse the video information of the original video from the video description file of the original video; Decoding the video encoding data to obtain a current video, and collecting statistics on a definition parameter, encoding parameter, current resolution parameter, and image quality distortion index of the current video to obtain video information of the current video; Based on the video information of the original video and the video information of the current video, a video description file of the current video is generated, and the video description file of the current video and the video identifier of the current video are stored in correspondence.

12. The device according to claim 9, characterized in that The video information acquisition module is further used to calculate the mean square error according to the pixel values ​​of the current video and the original video, so as to obtain the image quality distortion index of the current video.

13. The device according to claim 9, characterized in that The video information acquisition module is also used to input multiple video frames in the original video into a picture quality assessment model based on video features for feature extraction, obtain visual features of each video frame, and determine the original picture quality score of the original video based on the visual features of each video frame.

14. The device according to claim 9, characterized in that The trained neural network-based video quality assessment model includes multiple feature extraction networks connected in sequence, and the image quality score acquisition module is used to input the video information of the current video and the video information of the original video into the first feature extraction network among the multiple feature extraction networks connected in sequence to perform feature extraction and obtain the first intermediate feature; starting from the second feature extraction network, the intermediate feature output by the previous feature extraction network is extracted through the current feature extraction network to obtain the corresponding intermediate feature, until the last feature extraction network outputs the video quality feature representation of the current video.

15. The device according to claim 9, characterized in that The device also includes a picture quality score fusion module, which is used to obtain the transmission network parameters corresponding to the original video and the object behavior data about the current video; based on the encoding parameters of the current video, the transmission network parameters and the object behavior data, the video quality of the current video is predicted to obtain a reference picture quality score of the current video; The reference image quality score of the current video and the current image quality score are integrated to obtain a target image quality score of the current video, where the target image quality score is used to characterize the degree of image quality degradation of the current video.

16. The device according to claim 15, characterized in that The image quality score fusion module is further used to call an image quality evaluation model based on video-related information to respectively determine the image quality scores corresponding to the encoding parameters of the current video, the transmission network parameters, and the object behavior data; The encoding parameters of the current video, the transmission network parameters, and the image quality scores corresponding to the object behavior data are integrated to obtain a reference image quality score of the current video.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images