Video experience quality evaluation method, electronic equipment and storage medium

Abstract

The invention provides a video experience quality evaluation method, electronic equipment and a storage medium, and the method comprises the steps: firstly receiving a video code stream sent by second equipment, and obtaining the transmission quality information and code stream quality information of the video code stream; and then obtaining an evaluation result of the video experience quality according to the transmission quality information and the code stream quality information, and in end-to-end user experience quality evaluation, integrating the transmission quality information and the code stream quality information, so that the final evaluation result of the video experience quality is closer to the real experience of the user.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of video transmission, and in particular, to a method for evaluating video experience quality, an electronic device, and a storage medium. Background Art

[0002] With the advent of the self-media era and the innovation of video technology, Real Time Communications (RTC) services are booming. Real-time applications such as online meetings, telemedicine, and online education have also attracted increasing attention. Users are no longer restricted by the venue and can establish one-to-one or one-to-many connections anytime and anywhere. They can record audio and video streams in real time through user terminals such as computers or smartphones and send them to other user terminals to achieve end-to-end interaction. However, during the video transmission process, due to the dynamic changes in network conditions, the real-time transmitted video is often damaged in different types and degrees, resulting in a decline in visual quality, such as decreased clarity, stuttering, and latency.

[0003] Currently, real-time communication systems mainly use the technology of adaptive bitrate to transmit videos, that is, according to the current network environment, the encoding bitrate of the video is dynamically adjusted on the premise of not damaging the user's viewing experience quality (Quality of Experience, QoE). However, how to measure the impact of encoding damage and transmission damage on the visual quality of the human eye is still an urgent problem to be solved. Summary of the Invention

[0004] The embodiments of the present application provide a method for evaluating video experience quality, an electronic device, and a storage medium, which can realize a video experience quality evaluation that is closer to the user's real experience.

[0005] In a first aspect, the embodiments of the present application provide a method for evaluating video experience quality, which is applied to a first device. The method includes:

[0006] Receiving a video bitstream sent by a second device;

[0007] Obtaining the transmission quality information and bitstream quality information of the video bitstream;

[0008] Obtaining an evaluation result of the video experience quality according to the transmission quality information and the bitstream quality information.

[0009] In a second aspect, the embodiments of the present application provide a method for evaluating video experience quality, which is applied to a second device. The method includes:

[0010] Obtaining a source video;

[0011] Performing encoding processing on the source video to obtain a video bitstream;

[0012] Send the video stream to a first device so that the first device obtains transmission quality information and stream quality information of the video stream, and enables the first device to obtain an evaluation result of the video experience quality according to the transmission quality information and the stream quality information.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0014] One or more processors;

[0015] A memory storing one or more programs thereon. When the one or more programs are executed by the one or more processors, the one or more processors implement the video experience quality evaluation method applied to the first device as described in the first aspect above; or implement the video experience quality evaluation method applied to the second device as described in the second aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the video experience quality evaluation method as described in the first aspect above; or implements the video experience quality evaluation method applied to the second device as described in the second aspect.

[0017] The video experience quality evaluation method, electronic device, and storage medium provided by the embodiments of the present application. The video experience quality evaluation method first receives a video stream sent by a second device, obtains transmission quality information and stream quality information of the video stream, and then obtains an evaluation result of the video experience quality according to the transmission quality information and the stream quality information. In the end-to-end user experience quality evaluation, the transmission quality information and the stream quality information are integrated, so that the final evaluation result of the video experience quality is closer to the user's real experience. Description of the Drawings

[0018] The drawings are used to provide a further understanding of the technical solutions of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solutions of the present invention, and do not constitute a limitation to the technical solutions of the present invention.

[0019] Figure 1 is a schematic flowchart of a video experience quality evaluation method provided by an embodiment of the present application;

[0020] Figure 2 is a schematic flowchart of a video experience quality evaluation method provided by an embodiment of the present application;

[0021] Figure 3 is Figure 2 a sub-step flowchart of step S210 in;

[0022] Figure 4 isFigure 2 Schematic diagram of the sub-step process of step S220 in

[0023] Figure 5 Is Figure 4 Schematic diagram of the sub-step process of step S420 in

[0024] Figure 6 Is Figure 5 Schematic diagram of the sub-step process of step S520 in

[0025] Figure 7 Schematic diagram of the process of the training step of the experience quality prediction network provided by the embodiment of the present application;

[0026] Figure 8 Schematic diagram of the process of a video experience quality evaluation method provided by another embodiment of the present application;

[0027] Figure 9 Schematic diagram of the process of a video experience quality evaluation method provided by another embodiment of the present application;

[0028] Figure 10 Schematic diagram of the process of a video experience quality evaluation method provided by another embodiment of the present application;

[0029] Figure 11 Schematic diagram of the network structure of the experience quality prediction network provided by the embodiment of the present application;

[0030] Figure 12 Schematic diagram of the device structure of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0031] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] It should be understood that in the description of the embodiments of the present application, if there is a description of "first", "second", etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features. "At least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can indicate the situation of A existing alone, A and B existing simultaneously, and B existing alone. Wherein A and B can be singular or plural. The character " / " generally indicates that the front and rear associated objects are in an "or" relationship. "At least one of the following" and its similar expressions refer to any group of these items, including any group of single items or plural items. For example, at least one of a, b, and c can indicate: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0033] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] With the advent of the self-media era and the innovation of video technology, real-time communication (RTC) services are booming. Real-time applications such as online meetings, telemedicine, and online education have also attracted more and more attention. Users are no longer restricted by the venue and can establish one-to-one or one-to-many connections anytime and anywhere, and record audio and video streams in real time through user terminals such as computers or smartphones, and send them to other user terminals to achieve end-to-end interaction. However, during the video transmission process, due to the dynamic changes in network conditions, the real-time transmitted video is often damaged in different types and degrees, resulting in a decline in visual quality, such as a decrease in clarity, stuttering, and latency.

[0035] Currently, real-time communication systems mainly use the technology of adaptive bitrate to transmit video, that is, according to the current network environment, dynamically adjust the encoding bitrate of the video on the premise of not damaging the quality of experience (QoE) of users. However, how to measure the impact of encoding damage and transmission damage on the visual quality of the human eye remains an urgent problem to be solved.

[0036] Based on this, the embodiments of the present application provide a method, an electronic device, and a storage medium for evaluating video experience quality, which can realize a video experience quality evaluation that is closer to the real experience of users.

[0037] An embodiment of the present application first proposes a method for evaluating the video experience quality applied to a first device. The first device, as the receiving end of the video stream, is used to receive and reorganize the video stream. Please refer to Figure 1 , Figure 1 which shows a schematic flowchart of a method for evaluating the video experience quality provided by an embodiment of the present application. As shown in Figure 1 , the method for evaluating the video experience quality includes but is not limited to steps S110 to S130.

[0038] Step S110: Receive the video stream sent by the second device.

[0039] It can be understood that, as the sender of the video stream, the second device acquires the source video and performs encoding processing on the source video. Then, the second device sends the video stream obtained by encoding processing to the first device based on a specific transmission protocol. Among them, the source video can be a video collected by the second device through an audio-visual recording device, or the source video can be a video obtained by the second device by reading a video file or other means.

[0040] It should be noted that the type of encoding algorithm adopted can be determined according to the actual application scenario, such as H.264, H.265, AVS2, AVS3, AV1, H.266, etc. The embodiments of the present application do not make specific limitations on this.

[0041] Step S120: Obtain the transmission quality information and the bitstream quality information of the video stream.

[0042] It should be understood. Please refer to Figure 9 , Figure 9 which shows a schematic flowchart of the method for evaluating the video experience quality provided by an embodiment of the present application. As shown in Figure 9 , as the receiving end, after receiving and reorganizing the video stream, the first device extracts the transmission quality information from the video stream and obtains the bitstream quality information by analyzing the video stream. The transmission quality information is the key quality indicator (KQI) of video transmission, and the bitstream quality information is the key quality indicator KQI of the video stream.

[0043] In some embodiments, the transmission quality information includes at least one of video delay information, playback rate, and playback freeze rate.

[0044] It should be noted that the video delay information refers to the time difference between the acquisition and playback of the same image; the playback rate is the result of dividing the playback frame rate by the acquisition frame rate. If the result is equal to 1, it is played at the normal rate. If the result is greater than 1, it is fast-forwarded. If the result is less than 1, it is slow-played; the playback stuttering rate is the result of dividing the total stuttering duration within a period of time by the total duration. Based on the transmission quality information such as video delay information, playback quantity, and playback stuttering rate, a video experience quality score can be obtained, and a more accurate evaluation result closer to the user's real experience can be obtained.

[0045] In some embodiments, the bitstream quality information includes at least one of video bit rate, video resolution, video coding type, video frame rate, bitstream quantization parameter, and bitstream motion vector information.

[0046] It should be noted that the video bit rate refers to the number of bits transmitted per second during video coding, usually measured in bits per second (bps); the video resolution refers to the number of pixels representing the image, usually expressed in the form of width × height. Common resolutions include 720p (1280x720 pixels), 1080p (1920x1080 pixels), 2K (2560×1440 pixels), and 4K (3840x2160 pixels); the video frame rate represents the number of image frames displayed per second, usually measured in frames per second (fps).

[0047] Quantization is a process used to reduce the amount of data in video coding. The bitstream quantization parameter (QuantizationParameter, QP) is a parameter that controls the quantization level during the quantization process. A higher bitstream QP indicates a deeper quantization process, resulting in greater data compression, and thus may also cause image quality loss. Therefore, by extracting bitstream features from the video bitstream to obtain the bitstream quantization parameter, not only can the compression damage caused by video coding to the image quality be evaluated using the bitstream quantization parameter, but also the computational complexity of video experience quality evaluation can be reduced compared to extracting the corresponding video experience quality features from the target video obtained after video decoding.

[0048] Video coding achieves data compression by detecting and recording the motion of objects between different frames. The bitstream motion vector information is used to describe the motion of objects between adjacent image frames. Therefore, by extracting bitstream features from the video bitstream to obtain the bitstream motion vector information, not only can the compression damage caused by video coding to the image quality be evaluated using the bitstream motion vector information, but also the computational complexity of video experience quality evaluation can be reduced compared to extracting the corresponding video experience quality features from the target video obtained after video decoding.

[0049] In some specific embodiments, the bitstream quantization parameter is the quantization parameter of non-I frames in the video bitstream, and the bitstream motion vector information is the motion vector information of non-I frames in the video bitstream.

[0050] In the embodiments of the present application, first, image frame analysis is performed on the video bitstream to obtain the distribution of image frame types in the video bitstream, and then, according to the distribution of image frame types in the video bitstream, the quantization parameters and motion vector information of non-I frames in the video bitstream are obtained.

[0051] Exemplarily, in video compression coding, image frame types can be divided into I frames (Intra Frame), P frames (Predictive Frame), and B frames (Bidirectional Frame). Among them, an I frame refers to a key frame in a video sequence. Each video sequence usually starts with one or more I frames, which are used to provide a starting point for video decoding. An I frame contains complete image information and does not require reference to data of other frames for decoding; a P frame is a predictive frame and needs to refer to the previous frame for decoding. Therefore, a P frame only needs to store the difference information between the current frame and the reference frame to achieve the purpose of compressing the video; a B frame (Bidirectional Frame) is a bidirectional predictive frame and needs to refer to the frames before and after the current frame for decoding. A B frame stores the difference information between the current frame and the frames before and after it. That is to say, non-I frames refer to video frames that need to rely on data of other frames for decoding, such as P frames and B frames.

[0052] Step S130, obtain an evaluation result of the video experience quality according to the transmission quality information and the bitstream quality information.

[0053] It can be understood that after obtaining the transmission quality information and the bitstream quality information of the video bitstream, an evaluation result of the video experience quality is obtained according to the transmission quality information and the bitstream quality information, and a video experience quality evaluation result close to the user's subjective experience is given by comprehensively considering the transmission quality and the bitstream quality.

[0054] It should also be noted that, please refer to Figure 10 , Figure 10 which shows a schematic flowchart of a method for evaluating video experience quality provided by an embodiment of the present application. As Figure 10 shown, the first device serves as the receiving end of the video bitstream, and the second device serves as the sending end of the video bitstream. After the first device receives and reorganizes the video bitstream sent by the second device, it performs decoding processing on the video bitstream to obtain the target video. Then, the first device plays the target video through its own display module, or the first device transmits the target video to an external display device for video playback.

[0055] In some embodiments, please refer to Figure 2 , Figure 2 which shows a schematic flowchart of a method for evaluating video experience quality provided by an embodiment of the present application. As Figure 2As shown in the figure, the video experience quality evaluation method further includes step S210.

[0056] Step S210: Obtain image quality information, where the image quality information is the result of the image quality analysis of the video image corresponding to the video stream.

[0057] Correspondingly, obtaining the evaluation result of the video experience quality based on the transmission quality information and the stream quality information includes step S220.

[0058] Step S220: Obtain the evaluation result of the video experience quality based on the transmission quality information, the stream quality information, and the image quality information.

[0059] In the embodiments of the present application, the video experience quality evaluation method further includes obtaining the result of the image quality analysis of the video image corresponding to the video stream, that is, the image quality information, and then obtaining the evaluation result of the video experience quality based on the transmission quality information, the stream quality information, and the image quality information. On the basis of evaluating the video experience quality using the transmission quality information and the stream quality information, further considering the image quality before video encoding can obtain a more accurate video experience quality evaluation result that is closer to the user's real experience.

[0060] In some embodiments, the image quality information includes at least one of image blur information, image noise information, image distortion information, image color information, and image exposure accuracy information.

[0061] It should be noted that the image blur information is used to describe the unclear degree of the edges or details of the objects in the image; the image noise information is used to describe the random pixel changes in the image; the image distortion information is used to describe the distortion phenomenon of the shape or position of the objects in the image, such as radial distortion (bending caused by the lens), tangential distortion (deformation in the horizontal or vertical direction), etc.; the image color information is used to describe the color distribution and accuracy in the image; and the image exposure accuracy is used to characterize whether the light distribution in the image meets the expectation, and both under-exposure and over-exposure will affect the image quality.

[0062] In some embodiments, please refer to Figure 3 , Figure 3 shows Figure 2 the schematic diagram of the sub-step process of step S210 in Figure 3 As shown in the figure, obtaining the image quality information includes step S310 or step S320.

[0063] Step S310: Receive the image quality information sent by the second device.

[0064] Specifically, as Figure 9As shown in the figure, the second device serves as the sender of the video stream. After obtaining the target video, it can first perform image quality analysis on the target video to obtain the image quality information of each frame in the target video, then perform encoding processing on the target video to obtain the video stream, and finally send the image quality information and the video stream to the first device. Among them, the RTP / RTCP extension field can be used to transmit the image quality information, or the image quality information can be transmitted through the call protocol.

[0065] Step S320: Perform image quality analysis on the target video obtained after decoding the video stream to obtain image quality information.

[0066] Specifically, as Figure 10 shown, the first device serves as the receiver of the video stream. After receiving the video stream sent by the second device, it decodes the video stream to obtain the target video, and then performs image quality analysis on the target video to obtain the image quality information.

[0067] In a specific embodiment, first calculate the score of each index such as blur, noise, distortion, color, and exposure accuracy in the image, and then perform linear weighting on the scores of each index to obtain the image quality information.

[0068] Exemplarily, using a five-point scoring system, calculate the blur score, noise score, distortion score, color score, and exposure score in the image, and then perform linear weighting on the scores of each index to obtain the image quality score before video encoding. The image quality score = w1 * blur score + w2 * noise score + w3 * distortion score + w4 * color score + w5 * exposure score. Among them, the weighting coefficients w1, w2, w3, w4, and w5 can all take the value of 0.2. In addition, the weighting coefficients can also be adjusted according to different application scenarios.

[0069] In some embodiments, please refer to Figure 4 , Figure 4 shows Figure 3 the schematic flow diagram of the sub-steps of step S320 in Figure 4 shown, obtain the evaluation result of the video experience quality according to the transmission quality information, the stream quality information, and the image quality information, including but not limited to step S410 and step S420.

[0070] Step S410: Obtain the transmission quality information, the stream quality information, and the image quality information at different moments within the target time period of the video stream.

[0071] Step S420: Input the transmission quality information, the stream quality information, and the image quality information into the experience quality prediction network to obtain the evaluation result of the video experience quality within the target time period.

[0072] It can be understood that when evaluating the video experience quality, the transmission quality information, bitstream quality information, and image quality information at different moments within the target time period are collected, and the time-series transmission quality information, bitstream quality information, and image quality information are used as inputs to a pre-trained experience quality prediction network. The experience quality prediction network performs inference calculations based on the feature information of multiple time scales to obtain the evaluation result of the video experience quality within the target time.

[0073] In a specific embodiment, the input of the experience quality prediction network consists of 10 groups of parameters. Among them, 6 groups of parameters are bitstream quality information, namely video bit rate, video resolution, video coding type, video frame rate, bitstream quantization parameter, and bitstream motion vector information; 3 groups of parameters are transmission quality information, namely video delay information, playback rate, and freezing rate; and 1 group of parameters is image quality information. The input parameters of the experience quality prediction network are collected at regular time intervals, a total of N times. For example, the parameters are collected once every 1 second, and a total of 5 times are collected. If the video frame rate is high, the time interval can be adjusted down to 0.5 seconds or 0.25 seconds, etc., and the total number of collections can be adjusted to 10 times or 20 times, etc.

[0074] In some embodiments, please refer to Figure 5 , Figure 5 shows Figure 4 the schematic diagram of the sub-step process of step S420 in Figure 5 As shown, the transmission quality information, bitstream quality information, and image quality information of the video bitstream at different moments within the target time period are input into the experience quality prediction network to obtain the evaluation result of the video experience quality within the target time period, including but not limited to step S510 and step S520.

[0075] Step S510: Construct an experience quality feature matrix according to the transmission quality information, bitstream quality information, and image quality information.

[0076] Step S520: Input the experience quality feature matrix into the experience quality prediction network to obtain the evaluation result of the video experience quality within the target time period.

[0077] Please refer to Figure 11 , Figure 11 shows the schematic diagram of the network structure of an experience quality prediction network provided by an embodiment of the present application. As shown in Figure 11 , the transmission quality information, bitstream quality information, and image quality information form an experience quality feature matrix in the form of a time window as the input vector of the experience quality prediction network. The vector corresponding to each individual feature in the experience quality feature matrix is a value in multiple time dimensions, and the experience quality prediction network performs inference calculations based on the feature information of multiple time scales.

[0078] Specifically, the meaning of each vector in the quality of experience feature matrix is as follows:

[0079] Res t1 Refers to the resolution of the time period where t1 is located. Exemplarily, for the four resolutions of 360p, 540p, 720p, and 1080p, they are respectively identified with the values 1, 2, 3, and 4;

[0080] Bri t1 Refers to the bitrate after video compression encoding in the time period where t1 is located, with the unit of kilobits per second (kbps);

[0081] ImgQ t1 Refers to the average image quality score in the time period where t1 is located. If there are multiple image frames in this time period, the arithmetic mean of the image quality scores of multiple image frames is taken;

[0082] Enc t1 Refers to the type of encoding algorithm. Exemplarily, for the six encoding algorithm types of H.264, H.265, AVS2, AVS3, AV1, and H.266, they are respectively represented by the values 1, 2, 3, 4, 5, and 6;

[0083] Fps t1 Refers to the total number of video frames per second in the time period where t1 is located, with the unit of frames per second;

[0084] QP t1 Refers to the average QP of non-I frames in the video bitstream in the time period where t1 is located;

[0085] MV t1 Refers to the average MV of non-I frames in the video bitstream in the time period where t1 is located;

[0086] Dealy t1 Refers to the delay in the time period where t1 is located, with the unit of milliseconds (ms), which is the average delay between the acquisition and playback of all image frames in the time period;

[0087] PlayR t1 Refers to the playback rate in the time period where t1 is located;

[0088] Stuck t1 Refers to the frame drop rate in the time period where t1 is located.

[0089] In the embodiments of the present application, a quality of experience feature matrix is constructed in a time window manner according to transmission quality information, bitstream quality information, and image quality information, enabling the quality of experience prediction network to predict the quality of experience evaluation based on features of multiple time scales and improving the accuracy of video quality of experience evaluation.

[0090] In some embodiments, please refer to Figure 6 , Figure 6 which shows Figure 5 the schematic diagram of the sub - step process of step S520 in Figure 6 As shown, the quality - of - experience feature matrix is input into the quality - of - experience prediction network to obtain the evaluation result of the video quality of experience within the target time period, including but not limited to step S610 and step S620.

[0091] Step S610: Input the quality - of - experience feature matrix into the long - short - term memory network layer in the quality - of - experience prediction network to obtain the temporal feature information.

[0092] Step S620: Input the temporal feature information into the fully - connected network layer in the quality - of - experience prediction network to obtain the evaluation result of the video quality of experience within the target time period.

[0093] It should be understood that as Figure 11 shown, the quality - of - experience prediction network includes a long - short - term memory network layer (Long Short - Term Memory, LSTM) and a fully - connected network layer. The quality - of - experience features are specifically input as input vectors into the LSTM for extracting temporal features, and the obtained temporal feature information is input into the fully - connected network layer. The fully - connected network layer maps the temporal feature information into a predicted value, that is, the video quality - of - experience evaluation score within the current time period.

[0094] In some embodiments, please refer to Figure 7 , Figure 7 which shows the schematic diagram of the training method of the quality - of - experience prediction network provided by the embodiments of the present application. Figure 7 As shown, the quality - of - experience prediction network is trained through steps S710 to S730.

[0095] Step S710: Select multiple video samples from the video dataset according to the spatial complexity and temporal complexity of the video to obtain a video sample set.

[0096] Step S720: Use the video sample set to perform video transmission simulation to obtain the video transmission simulation information of each video sample in the video sample set.

[0097] Step S730: Train the quality - of - experience prediction network based on the video transmission simulation information of the video samples and the subjective quality - of - experience evaluation.

[0098] It should be understood that during the training process of the quality of experience prediction network, video samples with different spatial complexities and time complexities are collected. After being transmitted under different network conditions, the quality of experience prediction network is trained according to the quality of experience prediction values output by the quality of experience prediction network and the subjective quality of experience scores, and the network parameters of the experience prediction network are updated until the gap between the quality of experience prediction values output by the quality of experience prediction network and the subjective quality of experience scores meets the preset conditions.

[0099] Specifically, for various video datasets, sampling is performed according to two dimensions of spatial perceptual information (SI) and temporal perceptual information (TI) to ensure that the selected video samples can be widely distributed in the ST-TI feature space, thereby ensuring the diversity of the video sample set.

[0100] Among them, SI is the spatial complexity of the video:

[0101]

[0102] The above formula means that for each video frame F n after performing Sobel filtering, then calculating the standard deviation of the filtered video frame, and finally selecting the maximum standard deviation as SI.

[0103] Among them, TI is the time complexity of the video:

[0104]

[0105] M n (i, j) = F n (i, j) - F n-1 (i, j);

[0106] The above formula means that first calculate the pixel difference M n between the same positions on two adjacent frames F n-1 n (i, j), then calculate the standard deviation of the difference frame image, and then select the maximum standard deviation as TI. (i, j), and then calculate the standard deviation of the difference frame image, and then select the maximum standard deviation as TI.

[0107] In the embodiments of the present application, publicly available network trace data can be collected first, the network bandwidth within a period of time can be intercepted, and according to the source of the network traces, they can be divided into three categories: 3G, 4G, and WiFi. In each category of network trace sets, screening is performed according to two parameters, the average value and the standard deviation of the bandwidth, to ensure that the selected network traces can cover different network conditions. Then, based on the WebRTC protocol framework, an end-to-end real-time communication test platform is built. Based on the selected network traces and network traffic control tools, different network environments are reproduced. The transmission process of real-time video is simulated through the test platform, and real-time video data with different coding damages and transmission damages is recorded in real time. Exemplarily, the source video is collected by a local computer through a virtual camera and the video bitstream is transmitted to the server in real time through the test platform, and the server then forwards the real-time video bitstream back to the local computer through the streaming media protocol, so that the local computer can synchronously display and record the video.

[0108] In the video transmission simulation, distribution data such as the sending bitrate and receiving bitrate, stuttering, frame loss rate, and interaction delay of the video samples are recorded, and the video transmission simulation information of each video sample in the video sample set is obtained, which serves as the basic data for analyzing the factors affecting the subjective experience quality of the video. Finally, based on the video transmission simulation information and subjective experience quality evaluation of the video samples, the experience quality prediction network is trained. Among them, the subjective experience quality evaluation is the subjective score given by the subject from the perspectives of video clarity, smoothness, interaction delay, etc. when watching the video at the sending end and receiving end of the video bitstream.

[0109] In a specific embodiment, the video transmission simulation information of the video sample includes transmission quality information, bitstream quality information, and image quality information. An experience quality feature matrix is constructed according to the transmission quality information, bitstream quality information, and image quality information. Then, the experience quality feature matrix is used as an input vector and input into the experience quality prediction network. Through the experience quality prediction network, the evaluation prediction value of the video experience quality of each video sample is obtained. Then, the evaluation prediction value of the video experience quality is compared with the subjective experience quality evaluation, and loss functions such as mean square error and mean absolute error are used to update the network parameters of the experience quality prediction network, so that the output of the experience quality prediction network is close enough to the subjective experience quality evaluation.

[0110] Please refer to Figure 8 , Figure 8 which shows a method for evaluating video experience quality provided by an embodiment of the present application. This method for evaluating video experience quality is applied to a second device, and the second device serves as the sending end of the video bitstream. As Figure 8 shown, this method for evaluating video experience quality includes but is not limited to steps S810 to S830.

[0111] Step S810, obtain the source video.

[0112] Step S820: Perform encoding processing on the source video to obtain a video bitstream.

[0113] Step S820: Send the video bitstream to the first device, so that the first device can obtain the transmission quality information and bitstream quality information of the video bitstream, and enable the first device to obtain an evaluation result of the video experience quality based on the transmission quality information and bitstream quality information.

[0114] It should be understood that the second device can be a user terminal with audio and video recording functions, such as a smart phone, a personal computer, or a smart watch, etc. The user terminal is used to record audio and video streams in real time. After the user terminal performs encoding processing on the source video, it sends the video bitstream obtained from the encoding processing to other user terminals; or the first device is a user terminal externally connected with an audio and video recording device, and the user terminal is only used to receive the audio and video streams recorded in real time by the audio and video recording device. After the user terminal performs encoding processing on the source video, it sends the video bitstream obtained from the encoding processing to other user terminals. The second device obtains the source video, then performs encoding processing on the source video to obtain a video bitstream, and finally sends the video bitstream to the first device, so that the first device can obtain the transmission quality information and bitstream quality information of the video bitstream, and enable the first device to obtain an evaluation result of the video experience quality based on the transmission quality information and bitstream quality information. In the end-to-end user experience quality evaluation, by integrating the transmission quality information and bitstream quality information, the final evaluation result of the video experience quality is closer to the user's real experience.

[0115] It should be noted that the video experience quality evaluation method applied to the second device provided in the embodiments of the present application is based on the same inventive concept as the video experience quality evaluation method applied to the first device in the above embodiments. The specific steps, beneficial effects, etc. in the video experience quality evaluation method provided in the embodiments of the present application can be referred to the descriptions in the above embodiments, and will not be elaborated herein.

[0116] In some embodiments, the video experience quality evaluation method further includes step S910.

[0117] Step S910: Perform image quality analysis on the source video to obtain image quality information.

[0118] Correspondingly, sending the video bitstream to the first device includes step S821.

[0119] Step S821: Send the video bitstream and the image quality information to the first device, so that the first device can obtain an evaluation result of the video experience quality based on the transmission quality information, bitstream quality information, and image quality information.

[0120] Such as Figure 9As shown, before encoding the source video, the second device analyzes the image quality of the source video to obtain image quality information. Then, the second device sends the video bitstream and the image quality information to the first device. After receiving the video bitstream and the image quality information, the first device obtains the transmission quality information and the bitstream quality information of the video bitstream. Then, according to the transmission quality information, the bitstream quality information, and the image quality information, the evaluation result of the video experience quality is obtained. In the video experience quality evaluation, fully considering the image quality information before video encoding, as well as the transmission quality information and the bitstream quality information of the video bitstream, can achieve a more accurate video experience quality evaluation that is closer to the user's real experience.

[0121] The following describes the video experience quality evaluation method provided by the embodiments of the present application through two specific examples.

[0122] Example 1

[0123] As Figure 9 shown, the sending end obtains the source video, analyzes the image quality of the source video to obtain image quality information. At the same time, the source video is encoded to obtain a video bitstream. Finally, the image quality information and the video bitstream are packaged and sent to the receiving end through the network; after receiving the image quality information and the video bitstream transmitted by the sending end, the receiving end decodes the reorganized video bitstream to obtain the target video, extracts the transmission quality information from the video bitstream, and obtains the bitstream quality information by analyzing the video bitstream. Then, according to the transmission quality information, the bitstream quality information, and the image quality information, the video experience quality is predicted to obtain the evaluation result of the video experience quality. In addition, after obtaining the target video, the receiving end displays and plays the target video.

[0124] Example 2

[0125] As Figure 10 shown, the sending end obtains the source video, encodes the source video to obtain a video bitstream, and then sends the video bitstream to the receiving end through the network; after receiving the video bitstream transmitted by the sending end, the receiving end decodes the reorganized video bitstream to obtain the target video, extracts the transmission quality information from the video bitstream, and obtains the bitstream quality information by analyzing the video bitstream. After obtaining the target video, the image quality of the target video is analyzed to obtain image quality information, and then according to the transmission quality information, the bitstream quality information, and the image quality information, the video experience quality is predicted to obtain the evaluation result of the video experience quality. In addition, after obtaining the target video, the receiving end displays and plays the target video.

[0126] The embodiments of the present application also provide an electronic device, as Figure 12 shown, the electronic device 1200 includes:

[0127] One or more processors 1210;

[0128] A memory 1220 stores one or more programs. When the one or more programs are executed by one or more processors 1210, the one or more processors 1210 implement a video experience quality evaluation method.

[0129] As a non-transitory network system, the memory 1220 can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory 1220 may include high-speed random access memory and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 1220 may optionally include memories 1220 that are remotely located relative to the processor 1210, and these remote memories 1220 can be connected to the processor 1210 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0130] The memory 1220 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1220 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1220 and are called by the processor 1210 to execute the methods of the embodiments of this application.

[0131] The processor 1210 can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0132] In some embodiments, the electronic device further includes:

[0133] An input / output interface for implementing information input and output;

[0134] A communication interface for implementing communication interaction between this device and other devices, which can implement communication through a wired method (such as USB, network cable, etc.) or through a wireless method (such as a mobile network, WIFI, Bluetooth, etc.);

[0135] A bus for transmitting information between various components of the device (such as the processor 1210, the memory 1220, the input / output interface, and the communication interface);

[0136] Among them, the processor 1210, the memory 1220, the input / output interface, and the communication interface can achieve communication connections with each other inside the device through a bus.

[0137] An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions for executing the video experience quality evaluation method provided in the embodiment of the present application.

[0138] An embodiment of the present application further provides a computer program product including a computer program or computer instructions. The computer program or computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device executes and implements the video experience quality evaluation method provided in the embodiment of the present application.

[0139] The system architecture and application scenarios described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art know that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0140] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. The non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. The volatile memory can include random access memory (RAM) or an external cache. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0141] Those of ordinary skill in the art will understand that all or some of the steps and systems disclosed in the above methods can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0142] Some embodiments of the present application have been illustrated above with reference to the accompanying drawings, which does not limit the scope of the rights of the present invention. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the present invention shall fall within the scope of the rights of the present application.

Claims

1. A method for evaluating video experience quality, which is applied to a first device. The method includes: Receiving a video bitstream sent by a second device; Obtaining transmission quality information and bitstream quality information of the video bitstream; Obtaining an evaluation result of video experience quality based on the transmission quality information and the bitstream quality information.

2. The method according to claim 1, wherein The method further includes: Obtaining image quality information, where the image quality information is an image quality analysis result of a video image corresponding to the video bitstream; The obtaining the evaluation result of video experience quality based on the transmission quality information and the bitstream quality information includes: Obtaining the evaluation result of video experience quality based on the transmission quality information, the bitstream quality information, and the image quality information.

3. The method according to claim 2, characterized in that, The obtaining the image quality information includes: Receiving image quality information sent by the second device; Or, Performing image quality analysis on a target video obtained after decoding the video bitstream to obtain image quality information.

4. The method according to claim 2, wherein The obtaining the evaluation result of video experience quality based on the transmission quality information, the bitstream quality information, and the image quality information includes: Obtaining the transmission quality information, the bitstream quality information, and the image quality information of the video bitstream at different moments within a target time period; Inputting the transmission quality information, the bitstream quality information, and the image quality information into an experience quality prediction network to obtain an evaluation result of video experience quality within the target time period.

5. The method according to claim 4, wherein The inputting the transmission quality information, the bitstream quality information, and the image quality information into an experience quality prediction network to obtain an evaluation result of video experience quality within the target time period includes: Constructing an experience quality feature matrix based on the transmission quality information, the bitstream quality information, and the image quality information; Inputting the experience quality feature matrix into an experience quality prediction network to obtain an evaluation result of video experience quality within the target time period.

6. The method according to claim 5, wherein The inputting the experience quality feature matrix into an experience quality prediction network to obtain an evaluation result of video experience quality within the target time period includes: Inputting the experience quality feature matrix into a long short-term memory network layer in the experience quality prediction network to obtain temporal feature information; Inputting the temporal feature information into a fully connected network layer in the experience quality prediction network to obtain an evaluation result of video experience quality within the target time period.

7. The method according to claim 4, characterized in that, The experience quality prediction network is trained through the following steps: Selecting multiple video samples from a video dataset according to the spatial complexity and temporal complexity of the video to obtain a video sample set; Performing video transmission simulation using the video sample set to obtain video transmission simulation information of each video sample in the video sample set; Training the experience quality prediction network based on the video transmission simulation information of the video samples and subjective experience quality evaluation.

8. The method according to claim 1, characterized in that The transmission quality information includes at least one of video delay information, playback rate, and playback stutter rate.

9. The method according to claim 1, wherein The bitstream quality information includes at least one of video bit rate, video resolution, video coding type, video frame rate, bitstream quantization parameter, and bitstream motion vector information.

10. The method according to claim 2, wherein The image quality information includes at least one of image blurring information, image noise information, image distortion information, image color information, and image exposure accuracy information.

11. A video experience quality evaluation method, applied to a second device, the method comprising: Obtaining a source video; Performing encoding processing on the source video to obtain a video bitstream; Sending the video bitstream to a first device, so that the first device obtains transmission quality information and bitstream quality information of the video bitstream, and enables the first device to obtain an evaluation result of video experience quality according to the transmission quality information and the bitstream quality information.

12. The method according to claim 11, wherein The method further comprises: Performing image quality analysis on the source video to obtain image quality information; The sending the video bitstream to the first device includes: Sending the video bitstream and the image quality information to the first device, so that the first device obtains an evaluation result of video experience quality according to the transmission quality information, the bitstream quality information, and the image quality information.

13. An electronic device, comprising: One or more processors; A memory having stored thereon one or more programs, which when executed by the one or more processors cause the one or more processors to implement the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, having stored thereon a computer program, which when executed by a processor implements the method according to any one of claims 1 to 12.

Images