Video quality problem determination method, apparatus, storage medium, and program product

Abstract

The application relates to a video quality problem determination method and device, a storage medium and a program product. The video quality problem determination method comprises the following steps: receiving protocol information on a video player and a network device, determining an abnormal frame quality parameter and an abnormal packet quality parameter according to the protocol information, and determining a cause of video quality abnormality of the video player and a device according to the abnormal frame quality parameter and / or the abnormal packet quality parameter. According to the video quality problem determination method, the cause of the video quality problem and the device can be rapidly and accurately determined.

Description

Technical Field

[0001] This application relates to the field of video quality inspection, and in particular to a method, apparatus, storage medium, and program product for determining video quality problems. Background Technology

[0002] Compared to 4G, 5G technology has the advantages of high bandwidth and low latency, thus providing technical support for video backhaul scenarios that require real-time transmission of video streaming media, such as high-definition video live streaming, real-time camera monitoring, and drone shooting.

[0003] Video backhaul is a real-time business scenario based on streaming media. Taking high-definition video live streaming as an example, after video is captured by devices such as cameras, video frames are obtained. The video frames are encoded and compressed by encoders / backpack servers, etc. The compressed video frames are encapsulated by streaming media layer protocols to generate a live stream. The live stream is transmitted to the video server through network devices, and after decapsulation and decoding, the video player renders and displays the picture.

[0004] With the widespread application of video backhaul scenarios, a need has arisen to locate and analyze video quality issues in these scenarios. These issues include stuttering, screen tearing, and other problems that can negatively impact the user experience. These stuttering and tearing issues are caused by abnormal video frames, and multiple devices may be involved in the process of video frame acquisition, encoding, encapsulation, transmission, decapsulation, and decoding. To improve the user experience, it is necessary to pinpoint the device causing the video quality problems and the specific reasons why that device is causing the problems. Current technologies cannot quickly and accurately identify the cause of the video quality problems and the corresponding device. Summary of the Invention

[0005] In view of this, a method, apparatus, storage medium, and program product for determining video quality problems are proposed. The method, apparatus, storage medium, and program product for determining video quality problems according to the embodiments of this application can quickly and accurately determine the cause and device of the video quality problem.

[0006] In a first aspect, embodiments of this application provide a method for determining video quality problems. The method includes: receiving protocol information from a video playback terminal and a network device, wherein the network device is used to transmit video data to the video playback terminal, the protocol information indicating the transmission method of video frames, and the video frames including multiple data packets; determining abnormal frame quality parameters and abnormal packet quality parameters based on the protocol information, wherein the abnormal frame quality parameters indicate that the video frame to which the abnormal frame quality parameters belong has abnormal quality, and the abnormal packet quality parameters indicate that the data packets to which the abnormal packet quality parameters belong have abnormal quality; and determining the cause and device causing the video quality abnormality at the video playback terminal based on the abnormal frame quality parameters and / or the abnormal packet quality parameters.

[0007] According to the video quality problem determination method of this application, by receiving protocol information sent by the video playback end and protocol information sent by the network device used to transmit video data to the video playback end, the transmission method of the video frame and the multiple data packets included in the video frame can be determined based on the protocol information, thereby determining abnormal frame quality parameters and abnormal packet quality parameters. Since abnormal frame quality parameters indicate quality abnormalities in the video frame to which they belong, and abnormal packet quality parameters indicate quality abnormalities in the data packets to which they belong, and since video quality is correlated with video frames and data packets, abnormal frame quality parameters and abnormal packet quality parameters are associated with the occurrence of video quality abnormalities. Therefore, based on the abnormal frame quality parameters and abnormal packet quality parameters, the cause and device causing the video quality abnormality at the video playback end can be determined. The video quality problem determination method of this application, by jointly determining the cause and device causing the video quality abnormality at the video playback end through frame-level abnormal frame quality parameters and packet-level abnormal packet quality parameters, can reduce the time cost of determining the cause and device and improve the accuracy of the determined cause and device.

[0008] The video quality problem determination method of this application enables the extraction of protocol information from the payload head even when only the payload head can be obtained on certain devices, thus accurately determining the cause and device of the abnormal video quality at the video playback end, and expanding the scope of application of the video quality problem determination method.

[0009] According to the first aspect, in a first possible implementation of the video quality problem determination method, the protocol information includes streaming media layer protocol information, network layer protocol information, and transport layer protocol information of data packets. The streaming media layer protocol information includes streaming media layer protocol information of data packets on the video playback end. The network layer protocol information includes network layer protocol information of data packets on the network device and the video playback end. The transport layer protocol information includes transport layer protocol information of data packets on the network device and the video playback end. Determining abnormal frame quality parameters and abnormal packet quality parameters based on the protocol information includes: determining at least one video frame and each data packet corresponding to each frame in the at least one video frame based on the streaming media layer protocol information; determining the frame quality parameters of each frame in the at least one video frame based on the streaming media layer protocol information and the network layer protocol information; determining the packet quality parameters of each data packet corresponding to each frame in the at least one video frame based on the network layer protocol information and the transport layer protocol information; determining the abnormal frame quality parameters based on the frame quality parameters corresponding to each frame in the at least one video frame and a frame quality parameter threshold; and determining the abnormal packet quality parameters based on the packet quality parameters of each data packet corresponding to each frame in the at least one video frame and a packet quality parameter threshold.

[0010] In this way, frame quality parameters and packet quality parameters can be determined by combining different protocol information, thus reflecting the quality of video frames and the data packets within them. Whether frame quality parameters and packet quality parameters are abnormal can be determined separately using frame quality parameter thresholds and packet quality parameter thresholds, making the identification of abnormal frame quality parameters and packet quality parameters more convenient.

[0011] According to the first possible implementation of the first aspect, in the second possible implementation of the video quality problem determination method, the step of determining the abnormal frame quality parameter based on the frame quality parameters and frame quality parameter thresholds corresponding to each frame in the at least one video frame, and determining the abnormal packet quality parameter based on the packet quality parameters and packet quality parameter thresholds corresponding to each data packet in the at least one video frame, includes: for each frame of the at least one video frame, determining the frame quality parameter whose value is greater than the frame quality parameter threshold as the abnormal frame quality parameter; and for each frame of the at least one video frame, determining the packet quality parameter whose value is greater than the packet quality parameter threshold as the abnormal packet quality parameter.

[0012] This method allows for the identification of abnormal frame and packet quality parameters. By comparing parameter values ​​with thresholds, abnormal parameters can be determined accurately and quickly.

[0013] According to the first or second possible implementation of the first aspect, in the third possible implementation of the video quality problem determination method, the streaming media layer protocol information of the data packet includes a frame timestamp indicating the video frame to which the data packet belongs; the network layer protocol information of the data packet includes a packet time indicating the time of the data packet; the frame quality parameters include a frame start packet jitter parameter and a frame end packet jitter parameter; the frame start packet jitter parameter indicates the latency jitter of the first data packet corresponding to the video frame; and the frame end packet jitter parameter indicates the latency jitter of the last data packet corresponding to the video frame. Determining the frame quality parameters of each frame in the at least one video frame based on the streaming media layer protocol information and the network layer protocol information includes: for each frame of the at least one video frame, determining the video quality parameters based on the packet time of the data packet corresponding to that video frame. The arrival times of the first and last packets of a frame are calculated, where the first packet arrival time represents the arrival time of the first data packet corresponding to the video frame, and the last packet arrival time represents the arrival time of the last data packet corresponding to the video frame. Based on the first and last packet arrival times, the first packet interval and the last packet interval of adjacent video frames are calculated, where the first packet interval represents the arrival time difference of the first data packet corresponding to adjacent video frames, and the last packet interval represents the arrival time difference of the last data packet corresponding to adjacent video frames. Based on the frame timestamp, the frame interval of adjacent video frames is calculated. Based on the first packet interval, the frame interval, and the frame first packet jitter parameter of the previous video frame, the frame first packet jitter parameter of the current video frame is calculated. Based on the last packet interval, the frame interval, and the frame last packet jitter parameter of the previous video frame, the frame last packet jitter parameter of the current video frame is calculated.

[0014] In this way, the frame-first packet jitter parameter, which represents the latency jitter of the first data packet corresponding to the video frame, and the frame-tail packet jitter parameter, which represents the latency jitter of the last data packet corresponding to the video frame, can be calculated. While characterizing the video frame quality, this improves the approximation between the frame quality parameters and the actual jitter of the video frame.

[0015] According to any one of the first to third possible implementations of the first aspect, in the fourth possible implementation of the video quality problem determination method, the streaming media layer protocol information of the data packet includes a streaming media layer sequence number indicating the order of the data packet in its respective video frame; the network layer protocol information of the data packet includes a packet time indicating the time of the data packet; the frame quality parameters include frame packet loss parameters and frame out-of-order parameters, wherein the frame packet loss parameters indicate the loss status of data packets in the video frame, and the frame out-of-order parameters indicate the out-of-order status of data packets in the video frame; determining the frame quality parameters of each frame in the at least one video frame according to the streaming media layer protocol information and the network layer protocol information includes: for each frame of the at least one video frame, determining the frame packet loss parameters of the video frame according to the streaming media layer sequence number of the data packet corresponding to the video frame; for each frame of the at least one video frame, sorting the data packets corresponding to the video frame according to the chronological order of the packet times of the data packets corresponding to the video frame to obtain the sorting result corresponding to the video frame; and determining the frame out-of-order parameters of the video frame according to the sorting result and the streaming media layer sequence number of the data packets corresponding to the video frame.

[0016] In this way, frame loss parameters, which indicate the loss of data packets in a video frame, and frame out-of-order parameters, which indicate the out-of-order status of data packets in a video frame, can be calculated. This not only characterizes the quality of the video frame but also improves the flexibility in selecting frame quality parameters.

[0017] According to the fourth possible implementation of the first aspect, in the fifth possible implementation of the video quality problem determination method, the step of determining the frame packet loss parameters of each frame of the at least one video frame based on the streaming media layer sequence number of the data packet corresponding to the video frame includes: for each frame of the at least one video frame, determining the maximum value, minimum value, and number of streaming media layer sequence numbers of the data packet corresponding to the video frame based on the streaming media layer sequence number of the data packet corresponding to the video frame; determining the theoretical value of the number of data packets corresponding to the video frame based on the maximum value and minimum value of the streaming media layer sequence number; and determining the number of data packets corresponding to the video frame based on the video... The frame loss parameter of the video frame is determined by the difference between the theoretical value of the number of data packets corresponding to the frame and the number of streaming media layer sequence numbers of the data packets corresponding to the video frame. The frame out-of-order parameter of the video frame is determined based on the sorting result and the streaming media layer sequence number of the data packets corresponding to the video frame, including: when the value of the streaming media layer sequence number of the current data packet is greater than the streaming media layer sequence number of the data packets preceding the current data packet in the sorting result, the frame out-of-order parameter of the video frame is determined based on the minimum value of the sorting sequence number of the current data packet, the streaming media layer sequence number of the current data packet, and the streaming media layer sequence number of the data packets corresponding to the video frame.

[0018] In this way, the frame packet loss parameter can reflect whether data packets are lost in the video frame, and if so, the number of lost packets; the frame out-of-order parameter can reflect whether data packets are out of order in the video frame, and if so, the depth of the out-of-order data packets, thereby improving the accuracy of determining the quality of the video frame based on the frame quality parameters.

[0019] According to any one of the first to fifth possible implementations of the first aspect, in the sixth possible implementation of the video quality problem determination method, the packet quality parameters of each data packet corresponding to each frame in the at least one video frame are determined according to the network layer protocol information and transport layer protocol information of the data packet. The packet quality parameters include one or more of the following: a packet interval parameter indicating the time interval between the current data packet and the previous data packet; a packet jitter parameter indicating the delay jitter between the current data packet and the previous data packet; an out-of-order parameter indicating the out-of-order status of the data packet; a packet loss parameter indicating the packet loss status of the data packet; a retransmission parameter indicating the retransmission status of the data packet; a delay parameter indicating the delay status of the data packet and the corresponding response message; an IP packet fragmentation parameter indicating whether the data packet is fragmented; and a checksum error parameter indicating whether the checksum of the data packet is incorrect.

[0020] In this way, multiple packet quality parameters can be obtained to characterize packet quality. By acquiring multiple packet quality parameters, the packet quality determined based on the packet quality parameters is closer to the actual packet quality. This can improve the accuracy of subsequent determination of the causes of video quality problems and the equipment based on abnormal packet quality parameters and abnormal frame quality parameters.

[0021] According to the sixth possible implementation of the first aspect, in the seventh possible implementation of the video quality problem determination method, the step of determining the cause and device leading to the video quality abnormality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters includes: for any video frame, when the frame quality parameters of the video frame and the packet quality parameters of the data packet corresponding to the video frame satisfy a first condition, determining that the cause of the video quality abnormality at the video playback end is a video encoding problem, and determining that the device leading to the video quality abnormality at the video playback end is the device used for video encoding; wherein, the first condition includes: the frame packet loss parameters of the video frame are abnormal and the packet loss parameters of the data packet corresponding to the video frame are normal, or the frame out-of-order parameters of the video frame are abnormal and the out-of-order parameters of the data packet corresponding to the video frame are normal.

[0022] Since the protocol information of the data packets is obtained at the network device and the video playback end, the frame quality parameters and packet quality parameters can reflect abnormal situations during transmission. An abnormal frame packet loss parameter indicates that a data packet under the video frame is lost during transmission. If the packet loss parameter of the data packet under the video frame is normal at this time, it means that no data packet is lost during transmission. Therefore, it can be determined that the data packet was lost before transmission, that is, during video encoding. An abnormal frame out-of-order parameter indicates that a data packet under the video frame is out of order during transmission. If the out-of-order parameter of the data packet under the video frame is normal at this time, it means that the data packet is not out of order during transmission. Therefore, it can be determined that the data packet is out of order before transmission, that is, during video encoding. Based on the above reasons, the first condition, that is, the frame packet loss parameter of the video frame is abnormal and the packet loss parameter of the corresponding data packet of the video frame is normal, or the frame out-of-order parameter of the video frame is abnormal and the out-of-order parameter of the corresponding data packet of the video frame is normal, can accurately reflect video encoding problems.

[0023] According to the sixth or seventh possible implementation of the first aspect, in the eighth possible implementation of the video quality problem determination method, the step of determining the cause and device leading to the video quality abnormality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters includes: for any video frame, when the frame quality parameters of the video frame and the packet quality parameters of the data packet corresponding to the video frame satisfy a second condition, determining that the cause of the video quality abnormality at the video playback end is a terminal parameter problem, and determining that the device leading to the video quality abnormality at the video playback end is the device used for video playback; wherein, the second condition includes: any parameter among the frame start packet jitter parameters and frame end packet jitter parameters of the video frame is abnormal, and any parameter among the IP packet fragmentation parameters and checksum error parameters of the data packet corresponding to the video frame is abnormal.

[0024] Since the protocol information of the data packets is obtained at the network device and the video playback end, the frame quality parameters and packet quality parameters can reflect abnormalities in the transmission process. Abnormal frame start packet jitter parameters / frame end packet jitter parameters indicate that the first / last data packet under the video frame has excessive latency jitter during transmission. If the IP packet fragmentation parameters of the data packets under the video frame are abnormal at this time, it means that the data packets have been fragmented during transmission. Therefore, it can be determined that the parameter configuration of the video playback end fails to meet the requirements for data packets to be transmitted without fragmentation. If the checksum error parameters of the data packets under the video frame are abnormal at this time, it means that the data packets failed to successfully complete the verification with the video playback end during transmission. Therefore, it can be determined that the parameter configuration of the video playback end fails to meet the requirements for successful verification with the data packets. Based on the above reasons, the second condition, namely, that any parameter of the frame start packet jitter parameters and frame end packet jitter parameters of the video frame is abnormal, and any parameter of the IP packet fragmentation parameters and checksum error parameters of the data packets corresponding to the video frame is abnormal, can accurately reflect the terminal parameter problem.

[0025] According to any one of the sixth to eighth possible implementations of the first aspect, in the ninth possible implementation of the video quality problem determination method, the step of determining the cause and device leading to the abnormal video quality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters includes: for any video frame, when the frame quality parameters of the video frame and the packet quality parameters of the data packet corresponding to the video frame satisfy a third condition, determining that the cause of the abnormal video quality at the video playback end is a network transmission problem; wherein, the third condition includes: the frame packet loss parameters of the video frame are abnormal and the packet loss parameters of the data packet corresponding to the video frame are abnormal, or the frame out-of-order parameters of the video frame are abnormal and the out-of-order parameters of the data packet corresponding to the video frame are abnormal, or any one of the frame start packet jitter parameters and frame end packet jitter parameters of the video frame is abnormal and any one of the packet jitter parameters, packet interval parameters, retransmission parameters, and delay parameters of the data packet corresponding to the video frame is abnormal.

[0026] Since the protocol information of the data packets is obtained at the network device and the video playback end, the frame quality parameters and packet quality parameters can reflect abnormalities in the transmission process. Abnormal frame packet loss parameters indicate that data packets within the video frame are lost during transmission. If the packet loss parameters of the data packets within the video frame are abnormal at this time, it indicates that data packets have been lost during transmission, thus confirming a problem with network transmission. Abnormal frame out-of-order parameters indicate that data packets within the video frame are out of order during transmission. If the out-of-order parameters of the data packets within the video frame are abnormal at this time, it indicates that data packets have been lost during transmission, thus confirming a problem with network transmission. Based on the above reasons, the third condition, "the frame packet loss parameters of the video frame are abnormal and the packet loss parameters of the corresponding data packets of the video frame are abnormal, or the frame out-of-order parameters of the video frame are abnormal and the packet loss parameters of the corresponding data packets of the video frame are abnormal," is considered valid. The abnormal out-of-order parameters of the data packets accurately reflect network transmission problems. Abnormal frame start packet jitter parameters / frame end packet jitter parameters indicate that the first / last data packet in the video frame experiences excessive latency jitter during transmission. If the packet interval parameter of the data packets in the video frame is abnormal, it indicates that the time difference between the two data packets during transmission is too large, confirming a network transmission problem. If the packet jitter parameter of the data packets in the video frame is abnormal, it indicates that the data packets experience excessive latency jitter during transmission, confirming a network transmission problem. If the retransmission parameter of the data packets in the video frame is abnormal, it indicates that the data packets undergo multiple transmissions during transmission, confirming a network transmission problem. If the latency parameter of the data packets in the video frame is abnormal, it indicates that the data packets take too long to transmit, confirming a network transmission problem. Based on the above reasons, the third condition, "any one of the frame start packet jitter parameters and frame end packet jitter parameters of the video frame is abnormal, and any one of the packet jitter parameters, packet interval parameters, retransmission parameters, and latency parameters of the data packets corresponding to the video frame is abnormal," accurately reflects terminal parameter problems.

[0027] According to the ninth possible implementation of the first aspect, in the tenth possible implementation of the video quality problem determination method, after determining that the cause of the abnormal video quality at the video playback end is a network transmission problem, the step of determining the cause and device of the abnormal video quality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters further includes: determining the network layer sequence number of the data packet corresponding to the abnormal packet quality parameters based on the network layer protocol information of the data packet; determining the packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameters based on the transport layer protocol information of the data packet; receiving network topology data on the network device, and determining the transmission path of each data packet in the network based on the network topology data; finding data packets on each network device along the transmission path whose packet time, network layer sequence number, transport layer sequence number, and checksum all meet the conditions based on the transmission path of the data packet corresponding to the abnormal packet quality parameters; and determining the network device causing the abnormal video quality at the video playback end from each network device along the transmission path of the found data packet with the packet quality parameters that meet the conditions.

[0028] The packet quality parameters of data packets on each device can indicate the impact of that device on the transmission of the data packet. By finding the corresponding data packets in each network device on the transmission path, the found data packets can be used to statistically analyze the impact of different network devices on the transmission of data packets during the transmission process, and to determine the specific device that caused the abnormal video quality at the video playback end.

[0029] According to the tenth possible implementation of the first aspect, in the eleventh possible implementation of the video quality problem determination method, the conditions satisfied by the packet time, network layer sequence number, transport layer sequence number, and checksum include: the absolute value of the difference between the packet time and the packet time of the data packet corresponding to the abnormal packet quality parameter is less than a preset threshold, and the network layer sequence number, transport layer sequence number, and checksum are all the same as the network layer sequence number, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameter.

[0030] In this way, the same data packet can be found on different network devices along the transmission path. By analyzing the abnormal packet quality parameters of the same data packet on different devices along the transmission path, it can be deduced that the cause of the abnormal packet quality parameters on different devices is either the current device or one of its upstream devices. Further investigation of the abnormal packet quality parameters on the upstream device allows for accurate identification of the specific network device causing the video quality abnormality at the playback end, when the cause is a network transmission problem.

[0031] According to the tenth or eleventh possible implementation of the first aspect, in the twelfth possible implementation of the video quality problem determination method, when there are multiple data packets corresponding to the abnormal packet quality parameters, multiple sets of data packets that meet the conditions are found. Based on the packet quality parameters of the found data packets that meet the conditions, the network device causing the abnormal video quality at the video playback end is determined from each network device on the transmission path of the data packets. This includes: based on the packet quality parameters of multiple sets of data packets that meet the conditions, repeatedly determining the network device that may cause the abnormal video quality at the video playback end from each network device on the transmission path of the data packets; and statistically analyzing the network device that appears most frequently among the repeatedly determined network devices that may cause the abnormal video quality at the video playback end, and using it as the network device causing the abnormal video quality at the video playback end.

[0032] By repeatedly determining the packet quality parameters of multiple data packets on various network devices along the transmission path, the accuracy of identifying the network devices causing video quality anomalies can be improved. This reduces the probability of video quality anomalies occurring due to the same problem on the same device after implementing corresponding solutions for the identified network devices.

[0033] Secondly, embodiments of this application provide a video quality problem determination device, the device comprising: an information receiving module, configured to receive protocol information from a video playback terminal and a network device, the network device being configured to transmit video data to the video playback terminal, the protocol information indicating the transmission method of video frames, the video frames including multiple data packets; a parameter determination module, configured to determine abnormal frame quality parameters and abnormal packet quality parameters according to the protocol information, the abnormal frame quality parameters indicating that the video frame to which the abnormal frame quality parameters belong has abnormal quality, the abnormal packet quality parameters indicating that the data packet to which the abnormal packet quality parameters belong has abnormal quality; and an abnormality cause determination module, configured to determine the cause and device leading to the video quality abnormality at the video playback terminal according to the abnormal frame quality parameters and / or the abnormal packet quality parameters.

[0034] According to the second aspect, in a first possible implementation of the video quality problem determination device, the protocol information includes streaming media layer protocol information, network layer protocol information, and transport layer protocol information of data packets. The streaming media layer protocol information includes streaming media layer protocol information of data packets on the video playback end. The network layer protocol information includes network layer protocol information of data packets on the network device and the video playback end. The transport layer protocol information includes transport layer protocol information of data packets on the network device and the video playback end. Determining abnormal frame quality parameters and abnormal packet quality parameters based on the protocol information includes: determining at least one video frame and each data packet corresponding to each frame in the at least one video frame based on the streaming media layer protocol information; determining the frame quality parameters of each frame in the at least one video frame based on the streaming media layer protocol information and the network layer protocol information; determining the packet quality parameters of each data packet corresponding to each frame in the at least one video frame based on the network layer protocol information and the transport layer protocol information; determining the abnormal frame quality parameters based on the frame quality parameters corresponding to each frame in the at least one video frame and a frame quality parameter threshold; and determining the abnormal packet quality parameters based on the packet quality parameters of each data packet corresponding to each frame in the at least one video frame and a packet quality parameter threshold.

[0035] According to the first possible implementation of the second aspect, in the second possible implementation of the video quality problem determining device, the step of determining the abnormal frame quality parameter based on the frame quality parameters and frame quality parameter thresholds corresponding to each frame in the at least one video frame, and determining the abnormal packet quality parameter based on the packet quality parameters and packet quality parameter thresholds corresponding to each data packet in the at least one video frame, includes: for each frame of the at least one video frame, determining the frame quality parameter whose value is greater than the frame quality parameter threshold as the abnormal frame quality parameter; and for each frame of the at least one video frame, determining the packet quality parameter whose value is greater than the packet quality parameter threshold as the abnormal packet quality parameter.

[0036] According to the first or second possible implementation of the second aspect, in a third possible implementation of the video quality problem determining device, the streaming media layer protocol information of the data packet includes a frame timestamp indicating the video frame to which the data packet belongs; the network layer protocol information of the data packet includes a packet time indicating the time of the data packet; the frame quality parameters include a frame start packet jitter parameter and a frame end packet jitter parameter; the frame start packet jitter parameter indicates the latency jitter of the first data packet corresponding to the video frame; and the frame end packet jitter parameter indicates the latency jitter of the last data packet corresponding to the video frame. Determining the frame quality parameters of each frame in the at least one video frame based on the streaming media layer protocol information and the network layer protocol information includes: for each frame of the at least one video frame, determining the video quality parameters based on the packet time of the data packet corresponding to that video frame. The arrival times of the first and last packets of a frame are calculated, where the first packet arrival time represents the arrival time of the first data packet corresponding to the video frame, and the last packet arrival time represents the arrival time of the last data packet corresponding to the video frame. Based on the first and last packet arrival times, the first packet interval and the last packet interval of adjacent video frames are calculated, where the first packet interval represents the arrival time difference of the first data packet corresponding to adjacent video frames, and the last packet interval represents the arrival time difference of the last data packet corresponding to adjacent video frames. Based on the frame timestamp, the frame interval of adjacent video frames is calculated. Based on the first packet interval, the frame interval, and the frame first packet jitter parameter of the previous video frame, the frame first packet jitter parameter of the current video frame is calculated. Based on the last packet interval, the frame interval, and the frame last packet jitter parameter of the previous video frame, the frame last packet jitter parameter of the current video frame is calculated.

[0037] According to any one of the first to third possible implementations of the second aspect, in the fourth possible implementation of the video quality problem determination device, the streaming media layer protocol information of the data packet includes a streaming media layer sequence number indicating the order of the data packet in its respective video frame; the network layer protocol information of the data packet includes a packet time indicating the time of the data packet; the frame quality parameters include frame packet loss parameters and frame out-of-order parameters, wherein the frame packet loss parameters indicate the loss status of data packets in the video frame, and the frame out-of-order parameters indicate the out-of-order status of data packets in the video frame; determining the frame quality parameters of each frame in the at least one video frame according to the streaming media layer protocol information and the network layer protocol information includes: for each frame of the at least one video frame, determining the frame packet loss parameters of the video frame according to the streaming media layer sequence number of the data packet corresponding to the video frame; for each frame of the at least one video frame, sorting the data packets corresponding to the video frame according to the chronological order of the packet times of the data packets corresponding to the video frame to obtain a sorting result corresponding to the video frame; and determining the frame out-of-order parameters of the video frame according to the sorting result and the streaming media layer sequence number of the data packets corresponding to the video frame.

[0038] According to the fourth possible implementation of the second aspect, in the fifth possible implementation of the video quality problem determining device, the step of determining the frame packet loss parameters of each frame of the at least one video frame based on the streaming media layer sequence number of the data packet corresponding to the video frame includes: for each frame of the at least one video frame, determining the maximum value, minimum value, and number of streaming media layer sequence numbers of the data packet corresponding to the video frame based on the streaming media layer sequence number of the data packet corresponding to the video frame; determining the theoretical value of the number of data packets corresponding to the video frame based on the maximum value and minimum value of the streaming media layer sequence number; and determining the number of data packets corresponding to the video frame based on the video quality problem determining device. The frame loss parameter of the video frame is determined by the difference between the theoretical value of the number of data packets corresponding to the frame and the number of streaming media layer sequence numbers of the data packets corresponding to the video frame. The frame out-of-order parameter of the video frame is determined based on the sorting result and the streaming media layer sequence number of the data packets corresponding to the video frame, including: when the value of the streaming media layer sequence number of the current data packet is greater than the streaming media layer sequence number of the data packets preceding the current data packet in the sorting result, the frame out-of-order parameter of the video frame is determined based on the minimum value of the sorting sequence number of the current data packet, the streaming media layer sequence number of the current data packet, and the streaming media layer sequence number of the data packets corresponding to the video frame.

[0039] According to any one of the first to fifth possible implementations of the second aspect, in the sixth possible implementation of the video quality problem determination device, the packet quality parameters of each data packet corresponding to each frame in the at least one video frame are determined according to the network layer protocol information and transport layer protocol information of the data packet. The packet quality parameters include one or more of the following: a packet interval parameter indicating the time interval between the current data packet and the previous data packet; a packet jitter parameter indicating the delay jitter between the current data packet and the previous data packet; an out-of-order parameter indicating the out-of-order status of the data packets; a packet loss parameter indicating the packet loss status of the data packets; a retransmission parameter indicating the retransmission status of the data packets; a delay parameter indicating the delay status of the data packets and the corresponding response messages; an IP packet fragmentation parameter indicating whether the data packets are fragmented; and a checksum error parameter indicating whether the checksum of the data packets is incorrect.

[0040] According to the sixth possible implementation of the second aspect, in the seventh possible implementation of the video quality problem determination device, the step of determining the cause and device leading to the video quality abnormality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters includes: for any video frame, when the frame quality parameters of the video frame and the packet quality parameters of the data packet corresponding to the video frame satisfy a first condition, determining that the cause of the video quality abnormality at the video playback end is a video encoding problem, and determining that the device leading to the video quality abnormality at the video playback end is the device used for video encoding; wherein, the first condition includes: the frame packet loss parameters of the video frame are abnormal and the packet loss parameters of the data packet corresponding to the video frame are normal, or the frame out-of-order parameters of the video frame are abnormal and the out-of-order parameters of the data packet corresponding to the video frame are normal.

[0041] According to the sixth or seventh possible implementation of the second aspect, in the eighth possible implementation of the video quality problem determination device, the step of determining the cause and device leading to the video quality abnormality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters includes: for any video frame, when the frame quality parameters of the video frame and the packet quality parameters of the data packet corresponding to the video frame satisfy the second condition, determining that the cause of the video quality abnormality at the video playback end is a terminal parameter problem, and determining that the device leading to the video quality abnormality at the video playback end is the device used for video playback; wherein, the second condition includes: any parameter among the frame start packet jitter parameters and frame end packet jitter parameters of the video frame is abnormal and any parameter among the IP packet fragmentation parameters and checksum error parameters of the data packet corresponding to the video frame is abnormal.

[0042] According to any one of the sixth to eighth possible implementations of the second aspect, in the ninth possible implementation of the video quality problem determination device, the step of determining the cause and device leading to the abnormal video quality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters includes: for any video frame, when the frame quality parameters of the video frame and the packet quality parameters of the data packet corresponding to the video frame satisfy a third condition, determining that the cause of the abnormal video quality at the video playback end is a network transmission problem; wherein, the third condition includes: the frame packet loss parameters of the video frame are abnormal and the packet loss parameters of the data packet corresponding to the video frame are abnormal, or the frame out-of-order parameters of the video frame are abnormal and the out-of-order parameters of the data packet corresponding to the video frame are abnormal, or any one of the frame start packet jitter parameters and frame end packet jitter parameters of the video frame is abnormal and any one of the packet jitter parameters, packet interval parameters, retransmission parameters, and delay parameters of the data packet corresponding to the video frame is abnormal.

[0043] According to the ninth possible implementation of the second aspect, in the tenth possible implementation of the video quality problem determining device, after determining that the cause of the abnormal video quality at the video playback end is a network transmission problem, the step of determining the cause and device of the abnormal video quality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters further includes: determining the network layer sequence number of the data packet corresponding to the abnormal packet quality parameters based on the network layer protocol information of the data packet; determining the packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameters based on the transport layer protocol information of the data packet; receiving network topology data on the network device, and determining the transmission path of each data packet in the network based on the network topology data; finding data packets on each network device along the transmission path whose packet time, network layer sequence number, transport layer sequence number, and checksum all meet the conditions based on the transmission path of the data packet corresponding to the abnormal packet quality parameters; and determining the network device causing the abnormal video quality at the video playback end from each network device along the transmission path of the found data packet with the packet quality parameters that meet the conditions.

[0044] According to the tenth possible implementation of the second aspect, in the eleventh possible implementation of the video quality problem determination device, the conditions satisfied by the packet time, network layer sequence number, transport layer sequence number, and checksum include: the absolute value of the difference between the packet time and the packet time of the data packet corresponding to the abnormal packet quality parameter is less than a preset threshold, and the network layer sequence number, transport layer sequence number, and checksum are all the same as the network layer sequence number, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameter.

[0045] According to the tenth or eleventh possible implementation of the second aspect, in the twelfth possible implementation of the video quality problem determination device, when there are multiple data packets corresponding to the abnormal packet quality parameters, multiple sets of data packets that meet the conditions are found. Based on the packet quality parameters of the found data packets that meet the conditions, the network device causing the abnormal video quality at the video playback end is determined from each network device on the transmission path of the data packets. This includes: based on the packet quality parameters of multiple sets of data packets that meet the conditions, repeatedly determining the network device that may cause the abnormal video quality at the video playback end from each network device on the transmission path of the data packets; and statistically analyzing the network devices that may cause the abnormal video quality at the video playback end, the network device that appears most frequently is selected as the network device causing the abnormal video quality at the video playback end.

[0046] Thirdly, embodiments of this application provide a video quality problem determination apparatus, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute one or more of the video quality problem determination methods described in the first aspect or various possible implementations of the first aspect.

[0047] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement one or more of the video quality problem determination methods described in the first aspect or various possible implementations of the first aspect.

[0048] Fifthly, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in an electronic device, the processor in the electronic device executes one or more of the video quality problem determination methods described in the first aspect or various possible implementations of the first aspect.

[0049] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description

[0050] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0051] Figure 1 This illustrates an exemplary application scenario of the video quality problem determination method according to embodiments of this application.

[0052] Figure 2 This document illustrates an example of identifying data stream types according to an embodiment of this application.

[0053] Figure 3 This diagram illustrates an exemplary method for determining video quality problems according to an embodiment of this application.

[0054] Figure 4 This diagram illustrates an exemplary method for determining abnormal frame quality parameters and / or abnormal packet quality parameters according to embodiments of this application.

[0055] Figure 5 An example of a method for determining video frames and corresponding data packets according to an embodiment of this application is shown.

[0056] Figure 6 An example is shown of how the frame start packet jitter parameters and frame end packet jitter parameters are calculated according to an embodiment of this application.

[0057] Figure 7 An example is shown of how the frame packet loss parameter is calculated according to an embodiment of this application.

[0058] Figure 8 An example is shown of how the frame reorder parameter is calculated according to an embodiment of this application.

[0059] Figure 9 An example of a normal distribution probability curve generated according to an embodiment of this application is shown.

[0060] Figure 10 An example is shown of determining anomalies in the implementation parameters according to an embodiment of this application.

[0061] Figure 11 An exemplary schematic diagram illustrating the correspondence between abnormal frame quality parameters, abnormal packet quality parameters, and the causes and devices leading to abnormal video quality at the video playback end, according to embodiments of this application.

[0062] Figure 12 An example is shown of a specific network device that, according to an embodiment of this application, causes abnormal video quality at the video playback end.

[0063] Figure 13 A schematic diagram of an exemplary structure of a video quality problem determination apparatus according to an embodiment of this application is shown.

[0064] Figure 14 A schematic diagram of an exemplary structure of a video quality problem determination device according to an embodiment of this application is shown. Detailed Implementation

[0065] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0066] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0067] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0068] In existing technologies, to pinpoint the causes of video quality problems such as stuttering and screen tearing caused by abnormal video frames, a method is proposed that involves collecting a large amount of data from devices in the existing network (i.e., the network where the abnormal video frame resides), and then using the statistical distribution of big data to analyze the collected data to identify the causes of the video quality problems and corresponding device solutions. The specific implementation of this technical solution is as follows:

[0069] Data collection methods, including probe acquisition and device reporting, are used to obtain user experience data and asset management data from user devices (i.e., video playback devices), and network topology data is obtained from network devices through device reporting. User experience data includes the average video mean opinion score (vMOS), buffering duration, buffering frequency, and the number of times video quality is switched (e.g., the number of times the video display format is switched between 1080P, 720P, and 360P). User experience data characterizes the user's viewing experience and can be used to determine video playback quality. Asset management data includes user network access information. Asset management data characterizes the connection relationships between user devices and network devices. Network topology data characterizes the connection relationships between network devices. Based on asset management data and network topology data, the total number of users connected to each network device can be determined.

[0070] For each network device, the percentage of users with poor video playback quality on that device can be determined as the quality drop rate for that device. By statistically analyzing the quality drop rates of each network device and considering the connectivity between them, the distribution of quality drop rates across the network can be determined.

[0071] Based on the determined quality deviation rate and preset threshold for each network device, the quality of experience (QoE) is determined. QoE includes calculating the coefficient of variation and mean of the quality deviation rate, and using it to identify problematic devices. For example, it involves determining the degree of skewness in the distribution of the coefficient of variation of the quality deviation rate relative to the preset threshold, or the degree of skewness in the distribution of the mean of the quality deviation rate relative to the preset threshold, and using the degree of skewness to find devices that cause significant deviations.

[0072] Based on the connection relationships between network devices, determine whether the network device has lower-level network devices. If not, the network device is the one identified by the existing technology as causing video quality problems. If it does, further confirm the cause and device of the video quality problem by examining the experience quality of the lower-level network devices.

[0073] Existing statistical distribution schemes utilizing big data have certain drawbacks. First, determining the cause and equipment responsible for a single video quality issue is time-consuming and lacks real-time performance. Second, these schemes require comprehensive analysis of data collected from multiple occurrences of the video quality problem. This means that the causes and equipment identified by existing technologies are specific to multiple occurrences of the problem, and these causes and equipment may differ across instances. Furthermore, the identified causes and equipment may only represent one possible outcome of multiple occurrences, and their accuracy is insufficient. Even optimizing the identified equipment based on the identified causes cannot reduce the probability of the same video quality problems recurring.

[0074] In conclusion, current technologies are not yet able to quickly and accurately determine the cause of video quality problems and identify the relevant equipment.

[0075] To address the aforementioned technical problems, this application provides a method, apparatus, storage medium, and program product for determining video quality problems. The method for determining video quality problems in the embodiments of this application can quickly and accurately determine the causes and devices leading to video quality problems.

[0076] Figure 1 This illustrates an exemplary application scenario of the video quality problem determination method according to embodiments of this application.

[0077] like Figure 1 As shown, this scenario could be, for example, a live video streaming scenario. This scenario could include a video provider for providing video data, multiple network devices for transmitting video data, and a video player for displaying the video to the user. The video provider could be a camera or other device capable of capturing video data, or a device that can receive, store, and output video data. The video provider can encode the captured or received video data and encapsulate it into a live stream format according to a streaming media layer protocol. The encapsulated video data can be transmitted through network devices, which could include, for example, […]. Figure 1The system includes wireless base stations, bearer network equipment, core network equipment, and switches. Wireless base stations can be, for example, 5G base stations (next-generation NB, gNB). Core network equipment can be, for example, a unified distributed gateway (UDG), and switches can be, for example, switches using a specific deployment method (end of row, EoR). The video data is ultimately output to a video playback device, which can be, for example, a mobile phone, tablet, or laptop. On the video playback device, the live stream video data can be decapsulated, decoded, and rendered, allowing users to watch the video.

[0078] in, Figure 1 The application scenarios also include a terminal device (or server) for determining the cause of video quality problems and the device itself. This terminal device (or server) may be different from the video provider, network device, and video playback device mentioned above. This terminal device (or server) may include the video quality problem determination apparatus of this application embodiment, capable of executing the video quality problem determination method of this application embodiment.

[0079] The terminal device (or server) may further include a protocol information acquisition device for collecting protocol information from network devices and video playback terminals, and a topology data acquisition device for collecting network topology data from network devices. Alternatively, the video quality problem determination device, the protocol information acquisition device, and the topology data acquisition device may each be located on a separate terminal device or server, or any two of the three may be located on the same terminal device or server.

[0080] Those skilled in the art should understand that the application scenarios of the embodiments of this application are not limited to the devices described above. For example, there may be multiple video playback terminals connected to a network device, and for each video playback terminal's video quality problem, the video quality problem determination method of the embodiments of this application can be executed separately on the terminal device (or server).

[0081] In one possible implementation, with Figure 1 Taking the application scenario as an example, the protocol information acquisition device and the topology data acquisition device can respectively collect protocol information and network topology data. Specifically, when the video provider continuously outputs a live stream, the protocol information acquisition device can continuously collect protocol information from the video playback end and network devices. If the connection relationship of the network devices remains unchanged during this process, the topology data acquisition device only needs to collect network topology data once. Based on the protocol information from the video playback end and network devices, the video quality problem determination method of this application embodiment can determine the cause and device leading to abnormal video quality at the video playback end.

[0082] The following describes an exemplary method for collecting protocol information in the embodiments of this application.

[0083] In one possible implementation, protocol information acquisition can be based on existing technologies. For example, the protocol information acquisition device first acquires data streams from the video playback terminal and network device. These data streams may include video streams, or other types of data streams such as audio streams. The protocol information acquisition device can identify the acquired data streams in conjunction with streaming media protocols, extracting data packets of type video stream from the data streams. After determining that the data is of type video stream, the protocol information acquisition device can further extract streaming media layer protocol information, network layer protocol information, and transport layer protocol information from the data packets of type video stream according to streaming media layer protocols, network layer protocols, and transport layer protocols, thereby completing the acquisition of protocol information.

[0084] Protocol information can be collected by deploying probes, capturing network packets using computers or other devices, or utilizing the user tracking functions built into network devices. This example will focus on collecting protocol information using probes. Probes can possess deep packet inspection (DPI) capabilities, automatically identifying different types of data streams and acquiring protocol information.

[0085] A data packet may include a payload header and a payload, where the payload contains the actual data content. The payload header includes protocol information indicating the data packet transmission method. To ensure data security, when obtaining protocol information, the data packet may be encrypted, or only the payload header may be inspected. For example, for data packets on a video playback device, its streaming media layer payload header, network layer payload header, and transport layer payload header can be inspected, and the corresponding streaming media layer protocol information, network layer protocol information, and transport layer protocol information can be extracted. For data packets on other network devices, the network layer payload header and transport layer payload header can be inspected, and the corresponding network layer protocol information and transport layer protocol information can be extracted.

[0086] For example, for the streaming media layer payload, protocol information can be extracted by identifying the application type and protocol type. Service awareness (SA) technology can be used to identify the application type of a specific data stream. For instance, if a user experiences video quality anomalies while watching a live video stream using a particular application during a certain time period, the relevant data streams that need analysis can be retrieved from the data stream on the video playback device used by the user, based on user identity information, data packet time information, and / or application type. By default, data is retrieved from all data streams on the video playback device. For the retrieved data streams, existing protocol-defined decision-making methods can be used to identify the data stream type layer by layer, from the network layer, transport layer, streaming media layer, and application layer, determining the video streams present in the data streams and distinguishing their specific types to identify the video stream type containing video frames. Figure 2 This document illustrates an example of identifying data stream types according to an embodiment of this application.

[0087] like Figure 2 As shown, the protocols used in the network layer may include the Internet Protocol (IP), and the protocols used in the transport layer may include the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP). The protocols used in the streaming media layer may include Real-Time Streaming Protocol (RTSP), Real-Time Messaging Protocol (RTMP), Real-Time Transport Protocol (RTP), Real-Time Transport Control Protocol (RTCP), UDP-based Data Transfer Protocol (UDT), or Secure Reliable Transport (SRT), as well as MPEG-2 transport stream (MPEG2-TS). The protocols used in the application layer may include the H.264 video standard protocol or the H.265 video standard protocol.

[0088] See Figure 2Based on the protocols used in the transport layer, streaming media layer, and application layer, the supported video stream types can include: TCP+RTMP video stream, TCP+RTSP+RTP video stream, UDP+RTP video stream, UDP+RTP+MPEG2-TS video stream, UDP+SRT / UDT+MPEG2-TS video stream, and UDP+MPEG2-TS video stream. These protocols may also support control streams, such as... Figure 2 The TCP+RTMP control flow, TCP+RTSP control flow, UDP+RTCP control flow, etc., do not contain video frames.

[0089] For different video streams, protocol information can be obtained from the data packets within the video stream. This protocol information may include network layer protocol information, transport layer protocol information, streaming media layer protocol information, and the transport layer checksum (hereinafter referred to as checksum) calculated according to the standard checksum calculation method.

[0090] The following describes an exemplary method for collecting network topology data in the embodiments of this application.

[0091] For example, network topology data can be collected from network devices. Network topology data can include the topological connections or service paths of the network devices connected to the video playback terminal. The service path represents the connection relationships between network devices through which the service flow passes. This allows the upstream and downstream relationships of network devices to be determined based on the network topology data.

[0092] After acquiring protocol information and network topology data, the protocol information acquisition device and the topology data acquisition device can respectively transmit the acquired information or data to the video quality problem determination device. The video quality problem determination device uses the video quality problem determination method according to the embodiments of this application. The protocol information and network topology data can be used to determine the cause and device causing abnormal video quality at the video playback end.

[0093] Figure 3 This diagram illustrates an exemplary method for determining video quality problems according to an embodiment of this application.

[0094] like Figure 3 As shown, in one possible implementation, the video quality problem determination method according to the embodiments of this application includes:

[0095] S1, receiving protocol information from the video playback terminal and the network device, wherein the network device is used to transmit video data to the video playback terminal, the protocol information indicates the transmission method of the video frame, and the video frame includes multiple data packets;

[0096] S2, determine the abnormal frame quality parameters and abnormal packet quality parameters according to the protocol information. The abnormal frame quality parameters indicate that the video frame to which the abnormal frame quality parameters belong is of abnormal quality, and the abnormal packet quality parameters indicate that the data packet to which the abnormal packet quality parameters belong is of abnormal quality.

[0097] S3, determine the cause and device that caused the abnormal video quality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters.

[0098] According to the video quality problem determination method of this application, by receiving protocol information sent by the video playback end and protocol information sent by the network device used to transmit video data to the video playback end, the transmission method of the video frame and the multiple data packets included in the video frame can be determined based on the protocol information, thereby determining abnormal frame quality parameters and abnormal packet quality parameters. Since abnormal frame quality parameters indicate quality abnormalities in the video frame to which they belong, and abnormal packet quality parameters indicate quality abnormalities in the data packets to which they belong, and since video quality is correlated with video frames and data packets, abnormal frame quality parameters and abnormal packet quality parameters are associated with the occurrence of video quality abnormalities. Therefore, based on the abnormal frame quality parameters and abnormal packet quality parameters, the cause and device causing the video quality abnormality at the video playback end can be determined. The video quality problem determination method of this application, by jointly determining the cause and device causing the video quality abnormality at the video playback end through frame-level abnormal frame quality parameters and packet-level abnormal packet quality parameters, can reduce the time cost of determining the cause and device and improve the accuracy of the determined cause and device.

[0099] The video quality problem determination method of this application enables the extraction of protocol information from the payload head even when only the payload head can be obtained on certain devices, thus accurately determining the cause and device of the abnormal video quality at the video playback end, and expanding the scope of application of the video quality problem determination method.

[0100] For example, in step S1, protocol information can be obtained. In step S2, video frame identification and data packet extraction under the video frame can be achieved. For video quality issues such as screen tearing and stuttering, frame quality parameters can be defined to characterize the quality of the video frame. There can be multiple frame quality parameters, such as the frame start packet jitter parameter, frame end packet jitter parameter, frame packet loss parameter, and frame out-of-order parameter mentioned below. Different frame quality parameters can be pre-set with different calculation methods. Using the preset calculation method for the frame quality parameters, the values ​​of the frame quality parameters can be calculated based on the protocol information. Anomalies in the calculated frame quality parameter values ​​are then detected using frame quality parameter thresholds, thereby identifying frame quality parameters with abnormal values, i.e., abnormal frame quality parameters. Based on the abnormal frame quality parameters, the corresponding abnormal video frame and all data packets under that abnormal video frame can be further identified. Packet quality parameters can also be defined to characterize the quality of data packets. There can be multiple packet quality parameters, such as the packet interval parameter, packet jitter parameter, packet loss parameter, out-of-order parameter, retransmission parameter, latency parameter, IP packet fragmentation parameter, and checksum error parameter mentioned below. Different packet quality parameters can be pre-set with different calculation methods. By using the preset calculation methods for packet quality parameters, the values ​​of packet quality parameters can be calculated based on protocol information. Anomalies in the calculated values ​​of packet quality parameters are then detected by using packet quality parameter thresholds, thereby identifying packet quality parameters with abnormal values.

[0101] In step S3, the cause and device causing the video quality anomaly are determined based on whether the abnormal frame quality parameters of the abnormal video frame and the abnormal packet quality parameters of the data packets under the abnormal video frame meet corresponding conditions. There can be multiple conditions, such as the first, second, and third conditions mentioned below. The determined cause and device causing the video quality anomaly may differ depending on the conditions met. If the determined cause of the video quality anomaly uniquely identifies the device causing the anomaly, for example, if the cause is a video encoding problem, then the device causing the anomaly is uniquely identified as the device used for video encoding. Similarly, if the cause is a terminal parameter problem, then the device used for video playback, i.e., the video playback terminal, is uniquely identified. In this case, the video quality problem determination method according to the embodiments of this application has achieved the determination of the cause and device causing the video quality anomaly. If the determined cause of the video quality anomaly cannot uniquely identify the device causing the anomaly, for example, if the cause is a network transmission problem, since multiple network devices may be involved in the network transmission process, the device causing the anomaly cannot be uniquely identified. This application can identify abnormal data packets based on abnormal packet quality parameters, and find multiple data packets on multiple network devices participating in the network transmission process whose protocol information meets the conditions based on the protocol information of the abnormal data packets. By analyzing the packet quality parameters of the found multiple data packets, the device causing the abnormal video quality can be determined.

[0102] In the video quality problem determination method of this application embodiment, when determining the cause and device of the current video playback terminal video quality anomaly, the frame quality parameter threshold and packet quality parameter threshold used can be obtained based on the AI ​​model. The AI ​​model can be trained using frame quality parameters and packet quality parameters calculated over a period of time before determining the cause and device of the current video playback terminal video quality anomaly. This can solve the common problems of needing to label samples and the difficulty in obtaining data for modeling abnormal samples, and reduce the complexity of determining abnormal frame quality parameters and abnormal packet quality parameters.

[0103] As described above, abnormal frame quality parameters can be abnormal parameters within frame quality parameters, and abnormal packet quality parameters can be abnormal parameters within packet quality parameters. In step S2, frame quality parameters and packet quality parameters can be determined first based on the protocol information, and then abnormal frame quality parameters and abnormal packet quality parameters can be determined based on the frame quality parameters and packet quality parameters.

[0104] Figure 4This diagram illustrates an exemplary method for determining abnormal frame quality parameters and / or abnormal packet quality parameters according to embodiments of this application.

[0105] like Figure 4 As shown, in one possible implementation, the protocol information includes streaming media layer protocol information, network layer protocol information, and transport layer protocol information of data packets. The streaming media layer protocol information includes the streaming media layer protocol information of data packets on the video playback end. The network layer protocol information includes the network layer protocol information of data packets on the network device and the video playback end. The transport layer protocol information includes the transport layer protocol information of data packets on the network device and the video playback end. Step S2 includes:

[0106] S21, Based on the streaming media layer protocol information, determine at least one video frame and each data packet corresponding to each frame in the at least one video frame.

[0107] S22, determine the frame quality parameters of each frame in the at least one video frame according to the streaming media layer protocol information and the network layer protocol information; the frame quality parameters may include frame start packet jitter parameters, frame end packet jitter parameters, frame packet loss parameters, and frame out-of-order parameters.

[0108] S23, based on the network layer protocol information and the transport layer protocol information, determine the packet quality parameters of each data packet corresponding to each frame in the at least one video frame; the packet quality parameters may include packet interval parameters, packet jitter parameters, packet loss parameters, out-of-order parameters, retransmission parameters, delay parameters, IP packet fragmentation parameters, and checksum error parameters.

[0109] S24, determine the abnormal frame quality parameters based on the frame quality parameters and frame quality parameter thresholds corresponding to each frame in the at least one video frame, and determine the abnormal packet quality parameters based on the packet quality parameters and packet quality parameter thresholds corresponding to each data packet in the at least one video frame.

[0110] In this way, frame quality parameters and packet quality parameters can be determined by combining different protocol information, thus reflecting the quality of video frames and the data packets within them. Whether frame quality parameters and packet quality parameters are abnormal can be determined separately using frame quality parameter thresholds and packet quality parameter thresholds, making the identification of abnormal frame quality parameters and packet quality parameters more convenient.

[0111] Step S21 can be implemented based on existing technology. For example, in step S21, for the acquired protocol information, it is first classified according to the different types of video streams (see above). Figure 2(and related descriptions) to identify video frames and extract the corresponding data packets. A common method is to find data packets with streaming media type in the video stream based on the streaming media layer protocol information of the data packets (see step S211 below for an example). The streaming media type information is extracted from the streaming media layer protocol information of the found data packets with streaming media type (see step S212 below for an example). The streaming media layer sequence number is obtained from the streaming media layer protocol information of the data packets with streaming media type "video". Then, based on the streaming media layer sequence number, the data packets used to calculate the packet quality parameters and their frame timestamps are found. The frame timestamps of all data packets used to calculate the packet quality parameters are counted, and packets with the same frame timestamp are considered to belong to the same video frame (see step S213 below for an example). In this case, each frame in at least one video frame may include data packets with the same frame timestamp. The frame timestamps of data packets in different video frames may be different, thus allowing the determination of each video frame and its corresponding data packets.

[0112] For video streams carrying MPEG2-TS (UDP+RTP+MPEG2-TS video stream, UDP+SRT / UDT+MPEG2-TS video stream, UDP+MPEG2-TS video stream), the streaming media type can be determined by the value under Streamtype in the TS header of the protocol information. The streaming media layer sequence number can be determined by the Stream PID value when the value under Stream type in the protocol information is Video. For video streams carrying RTP other than those carrying MPEG2-TS (TCP+RTSP+RTP video stream, UDP+RTP video stream), the streaming media type can be determined by the value under Payload type in the protocol information, and the streaming media layer sequence number can be determined by the RTP SEQUENCENUMBER value when the value under Payload type is Video. For video streams carrying RTMP other than those carrying MPEG2-TS and RTP (TCP+RTMP video stream), the streaming media type can be determined by the value under Type ID in the protocol information, and the streaming media layer sequence number can be determined by the protocol information in the data packet with the value under Type ID being Video.

[0113] Figure 5 An example of a method for determining video frames and corresponding data packets according to an embodiment of this application is shown.

[0114] like Figure 5 As shown, taking a video stream of type UDP+RTP+MPEG2-TS as an example, the exemplary process of step S21 is as follows:

[0115] S211, in the obtained protocol information, first determine the data type based on the value of the TS PID field in the TS header. For example... Figure 5 The TS PID field value of 0 in TS 1# indicates that the data type of the TS transport segment to which this TS header belongs is a program association table (PAT). After determining that the data type of the TS transport segment is a PAT, the data packets with streaming media types are then determined based on the Program Map PID value in the TS header. For example, Figure 5 The Program Map PID field value of TS 1# is 0x0010f. In the protocol information, all TS transport segments that include the obtained Program Map PID value (0x0010f), for example... Figure 5 The TS transport segment corresponding to TS 8# in the protocol information is the data packet with streaming media type. The method described in step S211 above is performed on all TS headers in the streaming media layer protocol information to determine all data packets with streaming media type.

[0116] S212, from the TS header of the data packet with the determined streaming media type, first determine the streaming media type according to the Streamtype in the program mapping table. For example, as described in S211, Figure 5 TS 8# in the protocol header is the TS header for data packets containing streaming media type information. In TS 8#, the Stream type in the Program Map Table (PMT) is "Video," indicating that the streaming media type of the data packet to which TS 8# belongs is video. After determining that the streaming media type is video, the streaming media layer sequence number of the data packet with the video streaming media type is determined based on the Stream PID value in the PMT. For example, in TS 8#, the value of the Stream PID field in the PMT is 0x0100, which is the streaming media layer sequence number of the data packet to which TS 8# belongs. Protocol information includes data packets with the obtained Stream PID value (0x0100), which are data packets with the video streaming media type. For example... Figure 5 The data packets corresponding to TS 11#, TS 12#, TS 13#, and TS 14# are data packets whose streaming media type is video. For all data packets with the streaming media type, the method described in step S212 above is executed to find all data packets whose streaming media type is video.

[0117] S213, combine or encapsulate data packets of at least one TS transport segment of streaming media type video to obtain multiple data packets. After combination or encapsulation, each data packet includes multiple TS transport segments, for example, such as... Figure 5As shown, data packets corresponding to TS 11# and TS 12# are combined to obtain a single data packet. Data packets corresponding to TS 13# and TS 14# are combined to obtain a single data packet. The combination of data packets can be random. These combined or encapsulated data packets can be used to calculate packet quality parameters. Based on the protocol information, the data containing the Packetized Elementary Stream (PES) header is determined. The decode time stamp (DTS) is extracted from the PES header, and the unit of the DTS is converted to seconds. The value obtained after unit conversion is the frame timestamp of the data packet, that is, the frame timestamp of the video frame to which the data packet belongs. Since the identifier of a video frame is the frame timestamp of the streaming media layer, data packets with the same frame timestamp can be considered to belong to the same video frame. For all data packets of the streaming media type of video, the method described in step S213 above is executed to identify different video frames and determine the data packets corresponding to different video frames.

[0118] After step S21 identifies different video frames and determines their corresponding data packets, it is possible to further identify data packets with a streaming media layer. Frame timestamps and streaming media layer sequence numbers are obtained from the streaming media layer of the data packets. Transport layer sequence numbers, checksums, and acknowledgment message sequence numbers (ACK numbers) are obtained from the transport layer of the data packets. Network layer sequence numbers, fragmentation flags (Don't Fragment, DF), and packet timestamps are obtained from the network layer of the data packets. This information can be used in the process of calculating frame quality parameters and packet quality parameters.

[0119] Those skilled in the art will understand that at least one video frame may also include a portion of all video frames identified in step S21 (frames to which data packets of streaming media type video belong). It is possible to further configure the method to filter at least one video frame from the identified video frames based on whether the frame timestamps of the video frames are within the same time period, or to determine at least one video frame from the identified video frames based on the streaming media layer sequence number of the data packets in the identified video frames, or to filter at least one video frame from the identified video frames according to a pre-set sampling mechanism. As long as the streaming media type of the data packets in at least one video frame is video, this application does not limit the specific method of determining at least one video frame based on streaming media layer protocol information.

[0120] In step S22, for the different video frames identified in step S21, frame quality parameters for each video frame are obtained based on the streaming media layer protocol information and network layer protocol information of the data packets corresponding to the different video frames. The frame quality parameters of a video frame may include, for example, one or more of the following: frame start packet jitter parameters, frame end packet jitter parameters, frame packet loss parameters, and frame out-of-order parameters. By combining the frame quality parameters of the video frames with the packet quality parameters of the data packets corresponding to the video frames, the cause and device causing the video quality problem can be determined. For example, the cause of the video quality problem may be a video encoding problem, and the device causing the video quality problem may be the device used for video encoding; or, the cause of the video quality problem may be a video playback problem, and the device causing the video quality problem may be the device used for video playback; or, the cause of the video quality problem may be a network transmission problem, and the device causing the video quality problem may be a network device, etc. The calculation methods for different frame quality parameters are described below with examples.

[0121] Figure 6 An example is shown of how the frame start packet jitter parameters and frame end packet jitter parameters are calculated according to an embodiment of this application.

[0122] In one possible implementation, the streaming media layer protocol information of the data packet includes a frame timestamp indicating the video frame to which the data packet belongs. The network layer protocol information of the data packet includes a packet time indicating the time of the data packet. The frame quality parameters include a frame start packet jitter parameter and a frame end packet jitter parameter. The frame start packet jitter parameter indicates the latency jitter of the first data packet corresponding to the video frame, and the frame end packet jitter parameter indicates the latency jitter of the last data packet corresponding to the video frame.

[0123] Step S22 includes:

[0124] For each frame of the at least one video frame, the arrival time of the first packet and the arrival time of the last packet of the video frame are determined based on the packet time of the data packets corresponding to the video frame. The arrival time of the first packet represents the arrival time of the first data packet corresponding to the video frame, and the arrival time of the last packet represents the arrival time of the last data packet corresponding to the video frame.

[0125] Based on the arrival times of the first and last packets, the first packet interval and the last packet interval of adjacent video frames are calculated respectively. The first packet interval represents the arrival time difference of the first data packet corresponding to adjacent video frames, and the last packet interval represents the arrival time difference of the last data packet corresponding to adjacent video frames.

[0126] Calculate the frame interval between adjacent video frames based on the frame timestamp.

[0127] The frame first packet jitter parameters of the current video frame are calculated based on the first packet interval, frame interval, and frame first packet jitter parameters of the previous video frame.

[0128] The frame tail packet jitter parameters of the current video frame are calculated based on the tail packet interval, frame interval, and the frame tail packet jitter parameters of the previous video frame.

[0129] In this way, the frame-first packet jitter parameter, which represents the latency jitter of the first data packet corresponding to the video frame, and the frame-tail packet jitter parameter, which represents the latency jitter of the last data packet corresponding to the video frame, can be calculated. While characterizing the video frame quality, this improves the approximation between the frame quality parameters and the actual jitter of the video frame.

[0130] For example, the frame start packet jitter parameter and frame end packet jitter parameter in the embodiments of this application can be used to characterize the latency jitter of the data packets corresponding to the video frame. The larger the latency jitter of the data packets, the higher the probability of video screen tearing and stuttering.

[0131] After obtaining at least one video frame and the corresponding data packet according to step S21, the frame start packet jitter parameter and frame end packet jitter parameter can be calculated in the following way:

[0132] For example, with Figure 6 For example, at least one video frame may include four frames. For each video frame 1-4, based on the packet time (indicating the time of the data packet in the network layer protocol information of the data packet corresponding to that video frame), the data packet with the earliest packet time can be named the first packet, and the packet time of the first packet is the arrival time of the first packet. The data packet with the latest packet time is named the last packet, and the packet time of the last packet is the arrival time of the last packet. In this way, the first packet, the arrival time of the first packet, the last packet, and the arrival time of the last packet of each video frame in at least one video frame can be determined. Based on this, the arrival times of the first packets of video frames 1-4 can be recorded as T1, T2, T3, and T4, respectively, and the arrival times of the last packets of video frames 1-4 can be recorded as T1', T2', T3', and T4', respectively.

[0133] Based on the arrival times of the first and last packets of each video frame in video frames 1-4, calculate the first packet interval and the last packet interval of adjacent video frames. Where 2 ≤ n ≤ 4 and are integers, the first packet interval Deltan between video frame n and video frame n-1 can be calculated by the difference between the arrival times Tn and Tn-1 of the first packet of video frame n. The last packet interval Deltan' between video frame n and video frame n-1 can be calculated by the difference between the arrival times Tn and Tn-1' of the last packet of video frame n. When n = 1, the first packet interval Delta1 and the last packet interval Delta1' of video frame 1 are equal to 0 by default. Figure 6 For example, based on the arrival times T1, T2, T3, and T4 of the first packets of video frames 1-4, we can obtain the following: First packet interval Delta1 = 0 for video frame 1; First packet interval Delta2 between video frame 2 and video frame 1 = T2 - T1; First packet interval Delta3 between video frame 3 and video frame 2 = T3 - T2; First packet interval Delta4 between video frame 4 and video frame 3 = T4 - T3. Based on the arrival times T1', T2', T3', and T4' of the last packets of video frames 1-4, we can obtain the following: Last packet interval Delta1' = 0 for video frame 1; Last packet interval Delta2' between video frame 2 and video frame 1 = T2' - T1'; Last packet interval Delta3' between video frame 3 and video frame 2 = T3' - T2'; Last packet interval Delta4' between video frame 4 and video frame 3 = T4' - T3'.

[0134] For each video frame in video frames 1-4, the frame interval between adjacent video frames can be calculated based on the frame timestamp of the video frame to which the data packet belongs, as indicated in the network layer protocol information of the corresponding data packet. Figure 6 For example, the timestamps of video frames 1-4 can be recorded as TS1, TS2, TS3, and TS4, respectively. When 2 ≤ n ≤ 4 and are integers, the frame interval Intervaln between video frame n and video frame n-1 can be calculated by the difference between the timestamp TSn of video frame n and the timestamp TSn-1 of video frame n-1. When n = 1, the frame interval Interval1 of video frame 1 is equal to 0 by default. In this case, based on the timestamps TS1, TS2, TS3, and TS4 of video frames 1-4, we can obtain the following: the frame interval Interval1 of video frame 1 = 0; the frame interval Interval2 between video frame 2 and video frame 1 = TS2 - TS1; the frame interval Interval3 between video frame 3 and video frame 2 = TS3 - TS2; and the frame interval Interval4 between video frame 4 and video frame 3 = TS4 - TS3.

[0135] Based on the first packet interval and frame interval between each video frame in video frames 1-4 and its preceding video frame, and the frame first packet jitter parameter of the preceding video frame, the frame first packet jitter parameter of each video frame in video frames 1-4 can be calculated. The frame first packet jitter parameter of video frame 1 is equal to 0 by default. An example of how to calculate the frame first packet jitter parameter of other video frames besides video frame 1 is shown in formula (1):

[0136] Jittern=Jittern-1*15 / 16+(Deltan-Intervaln)*1 / 16 (1)

[0137] In formula (1), 2 ≤ n ≤ 4 and are integers. `Jittern` represents the frame start packet jitter parameter of video frame n. `15 / 16` and `1 / 16` are pre-set control parameters. The values ​​of the control parameters can be the optimal parameter values ​​determined by comparing the calculated frame start packet jitter parameter (or frame end packet jitter parameter) with the actual frame start packet jitter (or frame end packet jitter) using different values. The control parameters can be used to represent the proportion of the frame start packet jitter parameter (or frame end packet jitter parameter) of the previous video frame to the value of the frame start packet jitter parameter (or frame end packet jitter parameter) of the current video frame. Figure 6 Taking the first packet interval and frame interval of video frames 1-4 as an example, the frame first packet jitter parameters of video frames 1-4 can be obtained by combining formula (1) as follows:

[0138] Jitter1 = 0 for the first packet jitter of video frame 1;

[0139] The frame start packet jitter parameter Jitter2 of video frame 2 is Jitter1*15 / 16+(Delta2-Interval2)*1 / 16;

[0140] The frame start packet jitter parameter Jitter3 for video frame 3 is: Jitter2*15 / 16+(Delta3-Interval3)*1 / 16;

[0141] The frame start packet jitter parameter Jitter4 of video frame 4 is Jitter3*15 / 16+(Delta4-Interval4)*1 / 16.

[0142] Based on the tail packet interval and frame interval between each video frame in video frames 1-4 and its preceding video frame, and the tail packet jitter parameter of the preceding video frame, the tail packet jitter parameter of each video frame in video frames 1-4 can be calculated. The tail packet jitter parameter of video frame 1 is equal to 0 by default. An example of how to calculate the tail packet jitter parameter of other video frames besides video frame 1 is shown in formula (2):

[0143] Jittern'=Jittern-1'*15 / 16+(Deltan'-Intervaln')*1 / 16 (2)

[0144] In formula (2), 2 ≤ n ≤ 4 and are integers, Jittern' represents the frame tail packet jitter parameter of video frame n, and 15 / 16 and 1 / 16 are preset control parameters. Figure 6 Taking the end packet interval and frame interval of video frames 1-4 as an example, the frame end packet jitter parameters of video frames 1-4 can be obtained by combining formula (2) as follows:

[0145] Jitter1' = 0 for the frame end packet jitter parameter of video frame 1;

[0146] The frame tail packet jitter parameter Jitter2' of video frame 1 is: Jitter1'*15 / 16+(Delta2'-Interval2')*1 / 16;

[0147] The frame tail packet jitter parameter of video frame 1 is Jitter3' = Jitter2'*15 / 16 + (Delta3'-Interval3')*1 / 16;

[0148] The frame tail packet jitter parameter Jitter4' of video frame 1 is Jitter3'*15 / 16+(Delta4'-Interval4')*1 / 16.

[0149] In this way, the jitter parameters of the first packet and the last packet can be calculated.

[0150] Figure 7 and Figure 8 Examples of how frame packet loss parameters and frame out-of-order parameters are calculated according to embodiments of this application are shown respectively.

[0151] In one possible implementation, the streaming media layer protocol information of the data packet includes a streaming media layer sequence number indicating the order of the data packet within its respective video frame. The network layer protocol information of the data packet includes a packet time indicating the time of the data packet. The frame quality parameters include frame loss parameters and frame out-of-order parameters, whereby the frame loss parameters indicate the loss status of data packets in the video frame, and the frame out-of-order parameters indicate the out-of-order status of data packets in the video frame.

[0152] Step S22 includes:

[0153] For each frame of the at least one video frame, the frame packet loss parameters of the video frame are determined according to the streaming media layer sequence number of the data packet corresponding to the video frame.

[0154] For each frame of the at least one video frame, the data packets corresponding to the video frame are sorted according to the packet time sequence of the data packets corresponding to the video frame to obtain the sorting result corresponding to the video frame. Based on the sorting result and the streaming media layer sequence number of the data packets corresponding to the video frame, the frame disorder parameter of the video frame is determined.

[0155] In this way, frame loss parameters, which indicate the loss of data packets in a video frame, and frame out-of-order parameters, which indicate the out-of-order status of data packets in a video frame, can be calculated. This not only characterizes the quality of the video frame but also improves the flexibility in selecting frame quality parameters.

[0156] For example, frame packet loss can be determined from the perspective of the streaming media layer by statistically analyzing the streaming media layer sequence numbers. Figure 7 As shown, the streaming media layer sequence number S of the data packet ai corresponding to video frame 2 a -S i The possible values ​​are 11, 12, 13, 15, 16, 17, 18, 19, and 20, based on the statistically obtained streaming media layer sequence number S. a -S i The occurrence of frame packet loss can be analyzed and determined. An exemplary implementation can be found in the description below.

[0157] Frame out-of-order status can be determined from the perspective of the streaming media layer and network layer of data packets. This is achieved by sorting data packets according to their packet time, and then analyzing the sorting results along with the statistical streaming media layer sequence numbers of the data packets. For example, ... Figure 8 As shown, the streaming media layer sequence number S of the data packet AK corresponding to video frame 3 is... A -S K The data packets AK corresponding to video frame 3 are 21, 22, 23, 25, 26, 27, 28, 24, 29, 30, and 31, respectively. The sorting result of AK can be, for example, P = {A, B, C, D, E, F, G, H, I, J, K}. Based on the sorting result P and the statistically obtained streaming media layer sequence number S... A -S K The occurrence of frame packet loss can be analyzed and determined. An exemplary implementation can be found in the description below.

[0158] In one possible implementation, for each frame of the at least one video frame, the frame loss parameters of the video frame are determined based on the streaming media layer sequence number of the data packet corresponding to the video frame, including:

[0159] For each frame of the at least one video frame, based on the streaming media layer sequence number of the data packet corresponding to the video frame, determine the maximum value, minimum value, and number of streaming media layer sequence numbers of the data packet corresponding to the video frame. Based on the maximum and minimum values ​​of the streaming media layer sequence numbers, determine the theoretical value of the number of data packets corresponding to the video frame. Based on the difference between the theoretical value of the number of data packets corresponding to the video frame and the number of streaming media layer sequence numbers of the data packet corresponding to the video frame, determine the frame packet loss parameter of the video frame.

[0160] Based on the sorting result and the streaming media layer sequence number of the data packet corresponding to the video frame, the frame reorder parameters of the video frame are determined, including:

[0161] When the streaming media layer sequence number of the current data packet is greater than the streaming media layer sequence number of the data packets preceding the current data packet in the sorting result, the frame disorder parameter corresponding to the video frame is determined based on the minimum value of the sorting sequence number of the current data packet, the streaming media layer sequence number of the current data packet, and the streaming media layer sequence number of the data packet corresponding to the video frame.

[0162] In this way, the frame packet loss parameter can reflect whether data packets are lost in the video frame, and if so, the number of lost packets; the frame out-of-order parameter can reflect whether data packets are out of order in the video frame, and if so, the depth of the out-of-order data packets, thereby improving the accuracy of determining the quality of the video frame based on the frame quality parameters.

[0163] For example, such as Figure 7 As shown, the streaming media layer sequence number S of the data packet ai corresponding to video frame 2 a -S i The sequences are 11, 12, 13, 15, 16, 17, 18, 19, and 20. Based on the streaming media layer sequence number of the data packet ai corresponding to video frame 2, we can determine that the maximum value of the streaming media layer sequence number of data packet ai corresponding to video frame 2 is max = 20, the minimum value is min = 11, and the number of streaming media layer sequence numbers is 9. Based on the maximum value of the streaming media layer sequence number of video frame 2 (20) and the minimum value of 11, we can determine that the theoretical value of the number of data packets corresponding to video frame 2 is q = max - min + 1 = 20 - 11 + 1 = 10. The difference between q and the number of streaming media layer sequence numbers of the data packets corresponding to the video frame (9) is equal to 1. Therefore, there is packet loss in video frame 2, and the number of lost packets is 1.

[0164] like Figure 8 As shown, the streaming media layer sequence number S of the data packet AK corresponding to video frame 3 is... A -S KThe sequences are 21, 22, 23, 25, 26, 27, 28, 24, 29, 30, and 31, respectively. The sorting result could be, for example, P = {A, B, C, D, E, F, G, H, I, J, K}. Based on the streaming media layer sequence number S of the data packet AK corresponding to video frame 3... A -S K It can be determined that the streaming media layer sequence number (24) of data packet H is less than the streaming media layer sequence number (28) of data packet G, which precedes data packet H in the sorting result P. Therefore, it can be determined that video frame 3 is out of order, and the data packet causing the out-of-order situation is data packet H. The streaming media layer sequence number S of data packet H can be used to determine the order. H The minimum streaming media layer sequence number MIN = 21 for data packet AK corresponding to video frame 3 is used to calculate the theoretical value S of the sorting sequence number of data packet H in the sorting result P. H -MIN+1=4, and based on the sorting sequence number 8 of data packet H determined by the sorting result P and the theoretical value of the sorting sequence number 4 of data packet H in the sorting result P, the out-of-order depth 8-4=4 is calculated. Where there is only one data packet causing the out-of-order problem, the out-of-order depth of the out-of-order data packet can be used as the value of the frame out-of-order parameter corresponding to that video frame. When there are multiple data packets causing the out-of-order problem, the average out-of-order depth can be obtained by averaging the sum of the out-of-order depths of each out-of-order data packet based on the number of out-of-order data packets, and this average out-of-order depth is used as the value of the frame out-of-order parameter corresponding to that video frame.

[0165] In this way, frame packet loss parameters and frame out-of-order parameters can be calculated.

[0166] In step S23, for the different video frames identified in step S21, packet quality parameters are obtained based on the network layer protocol information and transport layer protocol information of the data packets corresponding to the different video frames. These parameters are used to reflect the data packet quality. The packet quality parameters may include, for example, one or more of the following: packet interval parameters, packet jitter parameters, out-of-order parameters, packet loss parameters, retransmission parameters, delay parameters, IP packet fragmentation parameters, and checksum error parameters. This process can be implemented based on existing technology. The following provides exemplary descriptions of different packet quality parameters and their calculation methods.

[0167] In one possible implementation, the packet quality parameters of each data packet corresponding to each frame in the at least one video frame are determined based on the network layer protocol information and transport layer protocol information of the data packet. The packet quality parameters include one or more of the following: a packet interval parameter indicating the time interval between the current data packet and the previous data packet; a packet jitter parameter indicating the delay jitter between the current data packet and the previous data packet; an out-of-order parameter indicating the out-of-order status of data packets; a packet loss parameter indicating the packet loss status of data packets; a retransmission parameter indicating the retransmission status of data packets; a delay parameter indicating the delay status of data packets; an IP packet fragmentation parameter indicating whether IP packet fragmentation exists in the data packet; and a checksum error parameter indicating whether the checksum of the data packet is incorrect.

[0168] In this way, multiple packet quality parameters can be obtained to characterize packet quality. By acquiring multiple packet quality parameters, the packet quality determined based on the packet quality parameters is closer to the actual packet quality. This can improve the accuracy of subsequent determination of the causes of video quality problems and the equipment based on abnormal packet quality parameters and abnormal frame quality parameters.

[0169] The calculation methods for each type of package quality parameter are described below.

[0170] The packet interval parameter can be determined based on the packet time in the network layer protocol information of the data packet. Its calculation method can be referenced from the example of frame interval calculation above. For example, assuming video frame 4 includes 5 data packets, the packet interval parameter can be calculated for each data packet (1-5) based on the packet time indicated in the network layer protocol information of that data packet. Specifically, when 2 ≤ m ≤ 5 and is an integer, the packet interval parameters for data packets m and m-1 can be calculated by the difference between the packet time of data packet m and the packet time of data packet m-1. When m = 1, the packet interval parameter for data packet 1 is equal to 0 by default.

[0171] Packet jitter parameters can be determined based on the packet time in the network layer protocol information of the data packets and preset control parameters. The calculation method can be referenced from the examples above regarding the calculation of frame-first packet jitter parameters or frame-last packet jitter parameters. For example, suppose at least one video frame includes video frames 1-4. For each video frame in video frames 1-4, the packet interval parameter can be calculated based on the packet time indicating the time of the data packet in the network layer protocol information of the corresponding data packet. Based on the packet interval parameter between each data packet and the previous data packet corresponding to each video frame, the packet jitter parameter of the previous data packet, and the preset control parameters, the packet jitter parameter of each data packet can be calculated. For example, for video frame 4, which may include 5 data packets, where 2 ≤ m ≤ 5 and is an integer, the packet jitter parameters of data packets m and m-1 can be determined by the packet interval parameter of data packets m and m-1, the packet jitter parameter of data packet m-1, and control parameters (e.g., 15 / 16 and 1 / 16).

[0172] The out-of-order parameter can be determined based on the network layer sequence number in the network layer protocol information and the transport layer sequence number in the transport layer protocol information of the current data packet and the previous data packet. When the current data packet is out of order, the value of the out-of-order parameter can be equal to 1; when the current data packet is not out of order, the value of the out-of-order parameter can be equal to 0. For example, if data packets are sorted according to packet time, and the transport layer sequence number and network layer sequence number of the current data packet are both lower than those of the previous data packet, then the current data packet is out of order.

[0173] Packet loss parameters can be determined based on the network layer sequence number in the network layer protocol information and the transport layer sequence number, checksum, and acknowledgment sequence number in the transport layer protocol information of the data packet. When the current data packet is lost, the value of the packet loss parameter can be equal to 1; when the current data packet is not lost, the value of the packet loss parameter can be equal to 0. For example, to calculate the packet loss parameters of data packets corresponding to a video stream using the TCP protocol, the data packets can be sorted according to their packet times. If, compared to the transport layer sequence number and network layer sequence number of the previous data packet, the transport layer sequence number of the current data packet decreases while the network layer sequence number increases, and before the packet time of the current data packet, no data packet with the same network layer sequence number, transport layer sequence number, and checksum as the current data packet appeared in the video buffer, but a data packet with the same acknowledgment sequence number as the current data packet appeared, then it indicates that the current data packet is lost. As another example, to calculate the packet loss parameters of data packets corresponding to a video stream using the UDP protocol, the network layer sequence number of the data packet can be used directly to determine whether a packet has been lost. Data packets can be sorted according to packet time. If the network layer sequence number of the current data packet decreases compared to the network layer sequence number of the previous data packet, it indicates that the current data packet is a lost data packet.

[0174] Retransmission parameters can be determined based on the network layer sequence number in the network layer protocol information and the transport layer sequence number, checksum, and acknowledgment sequence number in the transport layer protocol information of the data packet. When the current data packet is a retransmitted packet, the retransmission parameter value can be equal to 1; when the current data packet is not a retransmitted packet, the retransmission parameter value can be equal to 0. For example, to calculate the retransmission parameters of data packets corresponding to a video stream using the TCP protocol, the data packets can be sorted according to their packet times. If, compared to the transport layer sequence number and network layer sequence number of the previous data packet, the transport layer sequence number of the current data packet decreases while the network layer sequence number increases, and a data packet with the same network layer sequence number, transport layer sequence number, and checksum as the current data packet appeared in the video buffer before the packet time of the current data packet, then the current data packet is a retransmitted data packet. As another example, to calculate the retransmission parameters of data packets corresponding to a video stream using the UDP protocol, the data packets can be sorted according to their packet times. If a data packet exists with the same network layer sequence number and checksum as the current data packet, and its packet time is earlier than the packet time of the current data packet, then the current data packet is a retransmitted data packet.

[0175] The delay parameter can be determined based on the packet time in the network layer protocol information of the data packet and the acknowledgment sequence number in the transport layer protocol information. For example, to calculate the delay parameter of the data packet corresponding to the video stream using the TCP protocol, you can find the data packet with the same acknowledgment sequence number as the current data packet based on the acknowledgment sequence number of the current data packet, and determine the value of the delay parameter based on the difference between the packet time of the current data packet and the packet time of the found data packet.

[0176] IP packet fragmentation parameters can be determined based on the fragmentation flag in the network layer protocol information of the data packet. For example, if a fragmentation flag is present in the network layer protocol information, it means that the data packet is fragmented into IP packets, and the value of the IP packet fragmentation parameter can be equal to 1. If no fragmentation flag is present in the network layer protocol information, it means that the data packet is not fragmented into IP packets, and the value of the IP packet fragmentation parameter can be equal to 0.

[0177] The checksum error parameter can be determined based on the checksum in the transport layer protocol information of the data packet and the checksum value calculated using existing technology. If the two values ​​are inconsistent, it indicates that the current data packet has a checksum error. When a checksum error exists, the checksum error parameter value is equal to 1. When no checksum error exists, the checksum error parameter value is equal to 0. Notably, when the checksum value in the transport layer protocol information is invalid (value is 0), the checksum in the transport layer protocol information and the calculated checksum value are not compared.

[0178] In this way, a variety of packet quality parameters can be calculated. Those skilled in the art should understand that the types and calculation methods of packet quality parameters are not limited to the examples described above. As long as the packet quality parameters can reflect the quality of the data packet, the embodiments of this application do not limit the types and calculation methods of packet quality parameters.

[0179] In step S24, for the frame quality parameters calculated in step S22 and the packet quality parameters calculated in step S23, the anomalies of the parameters are determined by combining the frame quality parameter threshold and the packet quality parameter threshold, respectively.

[0180] In one possible implementation, step S24 includes:

[0181] For each frame of the at least one video frame, the frame quality parameters whose values ​​are greater than the frame quality parameter threshold are determined as the abnormal frame quality parameters.

[0182] For each frame of the at least one video frame, the packet quality parameters in the packet quality parameters of the data packets corresponding to that video frame whose values ​​are greater than the packet quality parameter threshold are determined as the abnormal packet quality parameters.

[0183] This method allows for the identification of abnormal frame and packet quality parameters. By comparing parameter values ​​with thresholds, abnormal parameters can be determined accurately and quickly.

[0184] For example, frame quality parameter thresholds and packet quality parameter thresholds can be preset as fixed values ​​or determined in real time using an AI model. The following describes an exemplary method for determining frame quality parameter thresholds and packet quality parameter thresholds using an AI model.

[0185] For example, each frame quality parameter can correspond to a frame quality parameter threshold; similarly, each packet quality parameter can correspond to a packet quality parameter threshold. When an AI model determines the frame quality parameter threshold (or packet quality parameter threshold) for a frame quality parameter (or packet quality parameter), the data used can be multiple values ​​of that frame quality parameter corresponding to multiple video frames (or multiple values ​​of that packet quality parameter corresponding to multiple data packets) calculated within a period prior to the current moment. For instance, using the frame first packet jitter parameter calculated within a period prior to the current moment (hereinafter referred to as the historical frame first packet jitter parameter) as the parameter sample for the AI ​​model, the AI ​​model can fit and generate a normal distribution probability curve P(x) of the historical frame first packet jitter parameter x based on the statistical situation of the historical frame first packet jitter parameter values, and use the cumulative distribution function F(a) (F(a)=P(x<=a)) of the normal distribution probability curve to calculate the frame quality parameter threshold a for the frame first packet jitter parameter. Figure 9 An example of a normal distribution probability curve generated according to an embodiment of this application is shown. Figure 9 As shown, the bar chart illustrates the distribution of jitter parameters in the first packet of historical frames, and the curve is the normal distribution probability curve P(x) fitted based on the distribution of jitter parameters in the first packet of historical frames.

[0186] An anomaly threshold is pre-set. When the cumulative distribution function F(a) equals the anomaly threshold, the value of 'a' can be the corresponding frame quality parameter threshold. If the frame first packet jitter parameter calculated after the current time step is greater than this threshold, it can be determined as an anomalous frame first packet jitter parameter, i.e., an anomalous frame quality parameter. Figure 9 Taking the normal distribution probability curve as an example, the anomaly threshold is preset to 75%. According to the cumulative distribution function of the normal distribution probability curve, when the cumulative distribution function F(a) is equal to 75%, the corresponding value of a can be equal to 100, for example. In this case, the normal range of the frame first packet jitter parameter is 0-100. Figure 10 Examples of abnormal condition determination of parameters according to embodiments of this application are shown. Figure 10As shown, the jitter parameter of the first packet of a certain frame that needs to be determined to be abnormal is 240, which is greater than 100. The jitter parameter of the first packet of this frame is an abnormal frame quality parameter, thus enabling the determination of abnormal parameter conditions.

[0187] Once the frame quality parameters (or packet quality parameters) for which anomalies need to be identified are calculated, the frame quality parameter thresholds (or packet quality parameter thresholds) obtained from the AI ​​model can be used to determine whether the frame quality parameters (or packet quality parameters) for which anomalies need to be identified are anomalous. Furthermore, parameter samples that have already been identified as anomalous frame quality parameters (or packet quality parameters) can be pruned from the parameter sample set, so that the frame quality parameter thresholds (or packet quality parameter thresholds) are also updated as the parameter samples change.

[0188] For example, the abnormal conditions of frame quality parameters can be determined using the method described above. The abnormal conditions of frame quality parameters can be shown in Table 1 below.

[0189] Table 1

[0190]

[0191] As shown in Table 1, a value of 1 for frame start packet jitter anomaly, frame end packet jitter anomaly, frame packet loss anomaly, and frame out-of-order anomaly indicates that the corresponding frame quality parameter of that frame is abnormal, and that frame is also an abnormal frame. According to Table 1, video frame 1 has no abnormal frame quality parameters; video frame 2 has abnormal frame start packet jitter and frame out-of-order parameters; video frame 3 has abnormal frame start packet jitter, frame end packet jitter, and frame packet loss parameters; and video frame 4 has no abnormal frame quality parameters.

[0192] The abnormal conditions of the package quality parameters are determined using the method described above. The abnormal conditions of the determined package quality parameters can be shown in Table 2 below.

[0193] Table 2

[0194]

[0195] In Table 2, gNB, UDG, and EoR represent a 5G base station, a unified distributed gateway, and a switch using a certain deployment method, respectively. A value of 1 for packet loss anomalies, out-of-order anomalies, packet jitter anomalies, packet interval anomalies, retransmission anomalies, latency anomalies, IP packet fragmentation anomalies, and checksum error anomalies indicates that the packet quality parameters are abnormal, and the packet is considered an abnormal packet. According to Table 2, on the 5G base station gNB, packets 1, 3, 4, and 5 have no abnormal packet quality parameters, while packet 2 has abnormal packet jitter parameters; on the unified distributed gateway UDG, packet 1 has abnormal packet jitter parameters, packet 2 has abnormal packet loss and latency parameters, and packet 3 has abnormal packet loss and jitter parameters; on the switch EoR using a certain deployment method, packet 1 has abnormal packet jitter parameters, packet 2 has abnormal packet loss parameters, and packet 3 has abnormal packet loss and jitter parameters.

[0196] After obtaining the abnormal frame quality parameters and abnormal packet quality parameters in step S2, in step S3, the cause and device causing the abnormal video quality at the video playback end can be determined based on the relationship between the abnormal frame quality parameters and abnormal packet quality parameters and preset conditions. Figure 11 An exemplary schematic diagram illustrating the correspondence between abnormal frame quality parameters, abnormal packet quality parameters, and the causes and devices leading to abnormal video quality at the video playback end, according to embodiments of this application.

[0197] like Figure 11As shown, the correspondence between video frames and data packets can be used to find the packet quality parameters corresponding to the frame quality parameters, ensuring that the data packet to which the packet quality parameter belongs is a data packet within the video frame to which the frame quality parameter belongs. Frame quality parameters and their corresponding packet quality parameters usually have a certain correlation. For example, if the packet loss parameter of a data packet is abnormal, the frame packet loss parameter of the video frame to which the data packet belongs can be checked. If the out-of-order parameter of a data packet is abnormal, the frame out-of-order parameter of the video frame to which the data packet belongs can be checked. If the packet interval parameter or packet jitter parameter of a data packet is abnormal, the data packet may not arrive at the expected decoding time, which may cause significant jitter in the video frame to which the data packet belongs; the frame start packet jitter parameter and frame end packet jitter parameter of the video frame to which the data packet belongs can be checked. If the retransmission parameter or delay parameter of a data packet is abnormal, the transmission of the data packet may slow down, which may cause significant jitter in the video frame to which the data packet belongs; the frame start packet jitter parameter and frame end packet jitter parameter of the video frame to which the data packet belongs can be checked. The IP packet fragmentation parameters and checksum error parameters of a data packet are also related to the jitter of the video frame to which the data packet belongs. Therefore, if the IP packet fragmentation parameters or checksum error parameters of a data packet are abnormal, you can check the jitter parameters of the first packet and the last packet of the video frame to which the data packet belongs.

[0198] As described above, there can be various abnormal frame quality parameters and abnormal packet quality parameters. We can first set conditions that correspond to the causes and devices that cause abnormal video quality at the video playback end, and then determine whether the abnormal frame quality parameters and abnormal packet quality parameters meet the corresponding conditions to determine the causes and devices that cause abnormal video quality at the video playback end.

[0199] The following is combined with Figure 11 The possible conditions that can be set, as well as the causes and devices that lead to abnormal video quality at the video playback end, as determined by the conditions, are described exemplarily.

[0200] In one possible implementation, step S3 includes:

[0201] For any video frame, if the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame satisfy the first condition, the cause of the abnormal video quality at the video playback end is determined to be a video encoding problem, and the device causing the abnormal video quality at the video playback end is determined to be the device used for video encoding.

[0202] The first condition includes:

[0203] The frame packet loss parameters of the video frame are abnormal, but the packet loss parameters of the data packets corresponding to the video frame are normal, or

[0204] The video frame has an abnormal frame order parameter, while the data packet corresponding to the video frame has a normal order parameter.

[0205] Since the protocol information of the data packets is obtained at the network device and the video playback end, the frame quality parameters and packet quality parameters can reflect abnormal situations during transmission. An abnormal frame packet loss parameter indicates that a data packet under the video frame is lost during transmission. If the packet loss parameter of the data packet under the video frame is normal at this time, it means that no data packet is lost during transmission. Therefore, it can be determined that the data packet was lost before transmission, that is, during video encoding. An abnormal frame out-of-order parameter indicates that a data packet under the video frame is out of order during transmission. If the out-of-order parameter of the data packet under the video frame is normal at this time, it means that the data packet is not out of order during transmission. Therefore, it can be determined that the data packet is out of order before transmission, that is, during video encoding. Based on the above reasons, the first condition, that is, the frame packet loss parameter of the video frame is abnormal and the packet loss parameter of the corresponding data packet of the video frame is normal, or the frame out-of-order parameter of the video frame is abnormal and the out-of-order parameter of the corresponding data packet of the video frame is normal, can accurately reflect video encoding problems.

[0206] For example, such as Figure 11 As shown, if the frame loss parameter in the frame quality parameters is abnormal and the packet loss parameter in the packet quality parameters is normal, or if the frame out-of-order parameter in the frame quality parameters is abnormal and the out-of-order parameter in the packet quality parameters is normal, it can be determined that the abnormal frame quality parameter and the abnormal packet quality parameter satisfy the first condition, and thus it can be determined that the cause of the video quality problem occurs in the video encoding process. The video encoding process is dominated by the device used for video encoding, and there is only one device used for video encoding. Therefore, it can be uniquely determined that the device causing the video encoding problem is the device used for video encoding.

[0207] In practical applications, it can be configured such that as long as it can be determined that the frame quality parameters and packet quality parameters satisfy any one of the first conditions, the cause and device causing the video quality abnormality at the video playback end can be directly determined, without needing to determine whether the frame quality parameters and packet quality parameters satisfy the other possibility of the first condition. Alternatively, it can be configured to determine whether the frame quality parameters and packet quality parameters satisfy each possibility of the first condition, and after determining that the frame quality parameters and packet quality parameters satisfy at least one possibility of the first condition, then determine the cause and device causing the video quality abnormality at the video playback end. Those skilled in the art should understand that both of the above settings can be used to determine the cause and device causing the video quality abnormality at the video playback end, and this application does not limit this.

[0208] In one possible implementation, step S3 includes:

[0209] For any video frame, if the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame satisfy the second condition, the cause of the abnormal video quality at the video playback end is determined to be a terminal parameter problem, and the device causing the abnormal video quality at the video playback end is determined to be the device used for video playback.

[0210] The second condition includes:

[0211] The video frame has an abnormal jitter parameter in either the frame start packet jitter parameter or the frame end packet jitter parameter, and the corresponding data packet of the video frame has an abnormal IP packet fragmentation parameter or checksum error parameter.

[0212] Since the protocol information of the data packets is obtained at the network device and the video playback end, the frame quality parameters and packet quality parameters can reflect abnormalities in the transmission process. Abnormal frame start packet jitter parameters / frame end packet jitter parameters indicate that the first / last data packet under the video frame has excessive latency jitter during transmission. If the IP packet fragmentation parameters of the data packets under the video frame are abnormal at this time, it means that the data packets have been fragmented during transmission. Therefore, it can be determined that the parameter configuration of the video playback end fails to meet the requirements for data packets to be transmitted without fragmentation. If the checksum error parameters of the data packets under the video frame are abnormal at this time, it means that the data packets failed to successfully complete the verification with the video playback end during transmission. Therefore, it can be determined that the parameter configuration of the video playback end fails to meet the requirements for successful verification with the data packets. Based on the above reasons, the second condition, namely, that any parameter of the frame start packet jitter parameters and frame end packet jitter parameters of the video frame is abnormal, and any parameter of the IP packet fragmentation parameters and checksum error parameters of the data packets corresponding to the video frame is abnormal, can accurately reflect the terminal parameter problem.

[0213] For example, such as Figure 11 As shown, if the frame quality parameters (first packet jitter / last packet jitter) are abnormal and the packet quality parameters (IP packet fragmentation / checksum error) are normal, it can be determined that the abnormal frame quality parameters and the abnormal packet quality parameters satisfy the second condition. Therefore, it can be determined that the cause of the video quality problem lies in the playback parameters during video playback. The video playback process is dominated by the device used for video playback, and there is only one device used for video playback (i.e., the video playback end). Therefore, it can be uniquely determined that the device causing the terminal parameter problem is the device used for video playback.

[0214] In practical applications, it can be configured such that as long as it can be determined that the frame quality parameters and packet quality parameters satisfy any one of the possibilities in the second condition, the cause and device causing the video quality abnormality at the video playback end can be directly determined, without having to determine whether the frame quality parameters and packet quality parameters satisfy the remaining possibilities in the second condition. Alternatively, it can be configured to determine whether the frame quality parameters and packet quality parameters satisfy each possibility in the second condition, and after determining that the frame quality parameters and packet quality parameters satisfy at least one possibility in the second condition, then determine the cause and device causing the video quality abnormality at the video playback end. Those skilled in the art should understand that both of the above settings can be used to determine the cause and device causing the video quality abnormality at the video playback end, and this application does not limit this.

[0215] In one possible implementation, step S3 includes:

[0216] For any video frame, if the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame satisfy the third condition, the cause of the abnormal video quality at the video playback end is determined to be a network transmission problem.

[0217] The third condition includes:

[0218] The video frame has abnormal frame packet loss parameters and the corresponding data packet has abnormal packet loss parameters, or

[0219] The video frame has an abnormal frame order parameter and the corresponding data packet has an abnormal order parameter, or

[0220] The video frame has an abnormal jitter parameter in either the frame start packet jitter parameter or the frame end packet jitter parameter, and the corresponding data packet has an abnormal jitter parameter, packet interval parameter, retransmission parameter, or delay parameter.

[0221] Since the protocol information of the data packets is obtained at the network device and the video playback end, the frame quality parameters and packet quality parameters can reflect abnormalities in the transmission process. Abnormal frame packet loss parameters indicate that data packets within the video frame are lost during transmission. If the packet loss parameters of the data packets within the video frame are abnormal at this time, it indicates that data packets have been lost during transmission, thus confirming a problem with network transmission. Abnormal frame out-of-order parameters indicate that data packets within the video frame are out of order during transmission. If the out-of-order parameters of the data packets within the video frame are abnormal at this time, it indicates that data packets have been lost during transmission, thus confirming a problem with network transmission. Based on the above reasons, the third condition, "the frame packet loss parameters of the video frame are abnormal and the packet loss parameters of the corresponding data packets of the video frame are abnormal, or the frame out-of-order parameters of the video frame are abnormal and the packet loss parameters of the corresponding data packets of the video frame are abnormal," is considered valid. The abnormal out-of-order parameters of the data packets accurately reflect network transmission problems. Abnormal frame start packet jitter parameters / frame end packet jitter parameters indicate that the first / last data packet in the video frame experiences excessive latency jitter during transmission. If the packet interval parameter of the data packets in the video frame is abnormal, it indicates that the time difference between the two data packets during transmission is too large, confirming a network transmission problem. If the packet jitter parameter of the data packets in the video frame is abnormal, it indicates that the data packets experience excessive latency jitter during transmission, confirming a network transmission problem. If the retransmission parameter of the data packets in the video frame is abnormal, it indicates that the data packets undergo multiple transmissions during transmission, confirming a network transmission problem. If the latency parameter of the data packets in the video frame is abnormal, it indicates that the data packets take too long to transmit, confirming a network transmission problem. Based on the above reasons, the third condition, "any one of the frame start packet jitter parameters and frame end packet jitter parameters of the video frame is abnormal, and any one of the packet jitter parameters, packet interval parameters, retransmission parameters, and latency parameters of the data packets corresponding to the video frame is abnormal," accurately reflects terminal parameter problems.

[0222] For example, such as Figure 11 As shown, if the frame quality parameters are abnormal and the packet quality parameters are also abnormal, or if the frame quality parameters are abnormal and the packet quality parameters are also abnormal, or if the frame quality parameters are abnormal and the packet quality parameters are also abnormal, or if the frame quality parameters are abnormal and the first packet jitter parameter / the last packet jitter parameter / the packet quality parameters are also abnormal, then it can be determined that the abnormal frame quality parameters and the abnormal packet quality parameters satisfy the third condition. Therefore, it can be determined that the cause of the video quality abnormality occurs during network transmission, and the cause of the video quality abnormality is the network device.

[0223] Table 3 shows an example of determining that the cause of the video quality problem is a network transmission problem according to the embodiments of this application.

[0224] Table 3

[0225]

[0226] Referring to Table 3, each frame timestamp uniquely identifies a frame. The packet information to be examined for a frame can include packet time, network layer sequence number, transport layer sequence number, checksum, and any anomalies in packet quality parameters. For abnormal packet quality parameters, the "Analysis Results" column will provide the analysis results indicating the video quality problem, based on the correlation between frame quality parameters and packet quality parameters. In the example in Table 3, since video frame 1 has abnormal packet loss parameters, and the corresponding data packet 2 and data packet 3 also have abnormal packet loss parameters, the third condition is met, confirming that the cause of the video quality problem is a network transmission issue.

[0227] In practical applications, it can be configured such that as long as it can be determined that the frame quality parameters and packet quality parameters satisfy any one of the possibilities in the third condition, the cause of the video quality anomaly at the video playback end can be directly determined, without having to determine whether the frame quality parameters and packet quality parameters satisfy the remaining possibilities in the third condition. Alternatively, it can be configured to determine whether the frame quality parameters and packet quality parameters satisfy each possibility in the third condition, and after determining that the frame quality parameters and packet quality parameters satisfy at least one possibility in the third condition, then determine the cause of the video quality anomaly at the video playback end. Those skilled in the art should understand that both of the above settings can be used to determine the cause of the video quality anomaly at the video playback end, and this application does not limit it.

[0228] Since multiple network devices may be used in the network transmission process, it is possible to further identify the specific device that caused the terminal parameter problem.

[0229] In one possible implementation, after determining that the cause of the abnormal video quality at the video playback end is a network transmission problem, step S3 further includes:

[0230] Based on the network layer protocol information of the data packet, determine the network layer sequence number of the data packet corresponding to the abnormal packet quality parameters;

[0231] Based on the transport layer protocol information of the data packet, determine the packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameters;

[0232] Receive network topology data from the network device, and determine the transmission path of each data packet in the network based on the network topology data;

[0233] Based on the transmission path of the data packets corresponding to the abnormal packet quality parameters in the network, find the data packets in each network device along the transmission path whose packet time, network layer sequence number, transport layer sequence number, and checksum all meet the conditions.

[0234] Based on the packet quality parameters of the data packets that meet the conditions, the network devices causing abnormal video quality at the video playback end are identified from among the network devices along the data packet transmission path.

[0235] The packet quality parameters of data packets on each device can indicate the impact of that device on the transmission of that data packet. By finding the corresponding data packets in each network device along the transmission path, the found data packets can be used to statistically analyze the impact of different network devices on the transmission of that data packet during the transmission process, and to determine the specific device that caused the abnormal video quality at the video playback end.

[0236] For example, after determining that the abnormal frame quality parameters and the abnormal packet quality parameters satisfy the third condition, the network layer sequence number of the data packet corresponding to the abnormal packet quality parameters can be determined according to the network layer protocol information. The packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameters can be determined according to the transport layer protocol information. Based on the network topology data collected by the topology data acquisition device, the transmission path of the data packet corresponding to the abnormal packet quality parameters on the network device can be obtained. For example, referring to Table 3, the abnormal packet quality parameters could be, for example, the packet loss parameters of data packet 2 and data packet 3. Based on the network topology data, the transmission path of data packets 2 and 3 could be, for example, gNB→UDG→EoR.

[0237] Based on the transmission path of the data packets corresponding to the abnormal packet quality parameters in the network, we can find data packets on each network device along the transmission path whose packet time, network layer sequence number, transport layer sequence number, and checksum all meet the conditions. By comparing the abnormal packet quality parameters of the data packets that meet the conditions, we can determine the network device causing the abnormal video quality at the video playback end. For example, referring to Table 3, we can find the data packets that meet the conditions based on the packet time (8:14:12.008), network layer sequence number (48160), transport layer sequence number (3480724955), and checksum (1231221433) of data packet 2, and based on the packet time (8:14:12.019), network layer sequence number (48162), transport layer sequence number (3480726183), and checksum (2134423523) of data packet 3.

[0238] In one possible implementation, the conditions satisfied by the packet time, network layer sequence number, transport layer sequence number, and checksum include:

[0239] The absolute value of the difference between the packet time and the packet time of the data packet corresponding to the abnormal packet quality parameter is less than a preset threshold, and the network layer sequence number, transport layer sequence number, and checksum are all the same as the network layer sequence number, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameter.

[0240] In this way, the same data packet can be found on different network devices along the transmission path. By analyzing the abnormal packet quality parameters of the same data packet on different devices along the transmission path, it can be deduced that the cause of the abnormal packet quality parameters on different devices is either the current device or one of its upstream devices. Further investigation of the abnormal packet quality parameters on the upstream device allows for accurate identification of the specific network device causing the video quality abnormality at the playback end, when the cause is a network transmission problem.

[0241] For example, combining Tables 2 and 3, based on the network layer sequence number (48160), transport layer sequence number (3480724955), and checksum (1231221433) of data packet 2 in Table 3, we can find data packets in Table 2 that satisfy the following conditions: network layer sequence number equals 48160, transport layer sequence number equals 3480724955, and checksum equals 1231221433. This allows us to find data packet 3 on the gNB, data packet 2 on the UDG, and data packet 2 on the EoR that meet the requirements. Further determination is needed to find... The absolute value of the difference between the packet time of the data packet and the packet time of data packet 2 in Table 3 (8:14:12.008) is less than a preset threshold. For example, the value of the preset threshold can be equal to 0.01. In this case, the absolute values ​​of the differences between the packet times of data packet 3 on the gNB, data packet 2 on the UDG, and data packet 2 on the EoR and data packet 2 in Table 3 are all less than the preset threshold. Data packet 3 on the gNB, data packet 2 on the UDG, and data packet 2 on the EoR can be data packets that meet the conditions found for data packet 2 in Table 3.

[0242] Based on the network layer sequence number (48162), transport layer sequence number (3480726183), and checksum (2134423523) of data packet 3 in Table 3, we can find data packets in Table 2 that satisfy the following conditions: network layer sequence number equals 48162, transport layer sequence number equals 3480726183, and checksum equals 2134423523. This allows us to find data packets 5 on the gNB, 3 on the UDG, and 3 on the EoR that meet the requirements. Further, we determine whether the absolute value of the difference between the packet time of the found data packets and the packet time (8:14:12.019) of data packet 3 in Table 3 is less than a preset threshold. For example, the preset threshold could be 0.01. In this case, the absolute values ​​of the differences between the packet times of data packets 5 on the gNB, 3 on the UDG, and 3 on the EoR and data packet 3 in Table 3 are all less than the preset threshold. Therefore, data packets 5 on the gNB, 3 on the UDG, and 3 on the EoR can be data packets that meet the conditions found for data packet 3 in Table 3.

[0243] Based on the packet quality parameters of the data packets that meet the criteria, the specific network device causing the abnormal video quality at the video playback end can be identified. Figure 12 An example is shown of a specific network device that, according to an embodiment of this application, causes abnormal video quality at the video playback end.

[0244] For example, such as Figure 12 As shown, A, B, and C represent gNB, UDG, and EoR, respectively. Taking the data packets on gNB, UDG, and EoR that meet the conditions for data packet 2 in Table 3 (hereinafter referred to as data packets on devices A, B, and C) as an example, according to the network topology data, the transmission path of the data packets can be determined to be gNB→UDG→EoR. The abnormal packet quality parameters of data packet 2 in Table 3 are packet loss parameters. When the packet loss parameters of data packets on devices A, B, and C are all abnormal, it can be determined that the packet loss is introduced by device A (gNB), which leads to abnormal video quality. When the packet loss parameters of data packets on B and C are abnormal, and the packet loss parameters of data packets on device A are normal, it can be determined that the packet loss is introduced by device B (UDG), which leads to abnormal video quality. When the packet loss parameters of data packets on device C are abnormal, and the packet loss parameters of data packets on devices A and B are normal, it can be determined that the packet loss is introduced by device C (EoR), which leads to abnormal video quality.

[0245] Theoretically, based on the gNB→UDG→EoR transmission path, if the packet loss parameters of device A are abnormal, then the packet loss parameters of devices B and C must also be abnormal; conversely, if the packet loss parameters of device B are abnormal, then the packet loss parameters of device C must also be abnormal. However, when collecting protocol information via probes, collection errors may occur, leading to inaccuracies in the protocol information and consequently, errors in the packet quality parameters determined based on that information. Therefore, when identifying the network device causing the video quality problem by finding data packets that meet the conditions on different network devices along the transmission path, the influence of protocol information collection errors can be further ruled out. For example, if the packet loss parameters of data packets on devices A and B are abnormal, but the packet loss parameters of data packets on device C are normal, it can be assumed that an error occurred when collecting protocol information from device C, thus determining that device A introduced packet loss, leading to abnormal video quality. Similarly, if the packet loss parameters of data packets on devices A and C are abnormal, but the packet loss parameters of data packets on device B are normal, it can be assumed that an error occurred when collecting protocol information from device B, thus determining that device A introduced packet loss, leading to abnormal video quality.

[0246] There are other examples of packet loss parameters on devices A, B, and C. For example, there are examples where the packet loss parameters on devices A, B, and C are all normal, where the packet loss parameters on device A are abnormal, and where the packet loss parameters on devices B and C are normal, where the packet loss parameters on device B are abnormal, and where the packet loss parameters on devices A and C are normal. Since these examples occur very rarely, they will not be described in detail here.

[0247] In this way, it is possible to distinguish between different situations such as protocol information collection errors on the device and packet loss during transmission, thereby improving the accuracy of identifying the device that causes abnormal video quality at the video playback end.

[0248] After comparing and analyzing using the above methods, the comparison results are shown in Table 4.

[0249] Table 4

[0250]

[0251] Refer to Table 4, where each packet (packet 2, packet 3) with abnormal packet quality parameters is uniquely identified by its network layer sequence number, transport layer sequence number, and checksum. The abnormal packet quality parameters are those that need to be compared with the found packets. Packet loss indicates the abnormality of the packet quality parameters that need to be compared for the packets on this device; 0 indicates that the packet quality parameter is normal, and 1 indicates that the packet quality parameter is abnormal. Combined with... Figure 11 It can be determined that the network device that caused the abnormalities in data packets 2 and 3 in Table 3, which in turn led to the abnormal video quality, is UDG.

[0252] In the example in Table 4, the network device identified as causing video quality anomalies is the same for different data packets. In other application scenarios, due to errors, multiple network devices may be identified as causing video quality anomalies for different data packets. In this case, the multiple identified network devices causing video quality anomalies can be further filtered to obtain the network device most likely causing the video quality anomalies.

[0253] In one possible implementation, when there are multiple data packets corresponding to the abnormal packet quality parameter, multiple sets of data packets that meet the conditions are found.

[0254] Based on the packet quality parameters of the found data packets that meet the criteria, the network devices causing the abnormal video quality at the video playback end are identified from among the network devices along the data packet transmission path, including:

[0255] Based on the packet quality parameters of multiple sets of data packets that meet the conditions, the network devices that may cause abnormal video quality at the video playback end are identified multiple times from each network device along the data packet transmission path.

[0256] The network device that appears most frequently among the network devices that are repeatedly identified as potentially causing video quality issues at the video playback end is considered the network device causing these issues.

[0257] By repeatedly determining the packet quality parameters of multiple data packets on various network devices along the transmission path, the accuracy of identifying the network devices causing video quality anomalies can be improved. This reduces the probability of video quality anomalies occurring due to the same problem on the same device after implementing corresponding solutions for the identified network devices.

[0258] For example, based on the frequency of occurrence of devices in the "Device Causing Anomalies" column under all abnormal video frames, the network device most likely to cause video quality anomalies can be identified. Referring to Table 4, UDG caused packet loss parameter anomalies twice, while other devices caused them zero times. Therefore, UDG is the network device most likely to cause video quality anomalies. In another example, if UDG caused packet loss parameter anomalies twice, EoR caused them five times, and gNB caused them zero times, then EoR is the network device most likely to cause video quality anomalies.

[0259] Figure 13 A schematic diagram of an exemplary structure of a video quality problem determination device according to an embodiment of this application is shown.

[0260] like Figure 13As shown, in one possible implementation, this application also proposes a video quality problem determination device, the device comprising: an information receiving module 130, configured to receive protocol information from a video playback terminal and a network device, the network device being configured to transmit video data to the video playback terminal, the protocol information indicating the transmission method of video frames, the video frames including multiple data packets; a parameter determination module 131, configured to determine abnormal frame quality parameters and abnormal packet quality parameters according to the protocol information, the abnormal frame quality parameters indicating that the video frame to which the abnormal frame quality parameters belong has abnormal quality, the abnormal packet quality parameters indicating that the data packets to which the abnormal packet quality parameters belong have abnormal quality; and an abnormality cause determination module 132, configured to determine the cause and device leading to the video quality abnormality at the video playback terminal according to the abnormal frame quality parameters and / or the abnormal packet quality parameters.

[0261] In one possible implementation, the protocol information includes streaming media layer protocol information, network layer protocol information, and transport layer protocol information for data packets. The streaming media layer protocol information includes the streaming media layer protocol information of data packets on the video playback end. The network layer protocol information includes the network layer protocol information of data packets on the network device and the video playback end. The transport layer protocol information includes the transport layer protocol information of data packets on the network device and the video playback end. Determining abnormal frame quality parameters and abnormal packet quality parameters based on the protocol information includes: determining at least one video frame and each data packet corresponding to each frame in the at least one video frame based on the streaming media layer protocol information; determining the frame quality parameters of each frame in the at least one video frame based on the streaming media layer protocol information and the network layer protocol information; determining the packet quality parameters of each data packet corresponding to each frame in the at least one video frame based on the network layer protocol information and the transport layer protocol information; determining the abnormal frame quality parameters based on the frame quality parameters corresponding to each frame in the at least one video frame and a frame quality parameter threshold; and determining the abnormal packet quality parameters based on the packet quality parameters of each data packet corresponding to each frame in the at least one video frame and a packet quality parameter threshold.

[0262] In one possible implementation, determining the abnormal frame quality parameter based on the frame quality parameters and frame quality parameter thresholds corresponding to each frame in the at least one video frame, and determining the abnormal packet quality parameter based on the packet quality parameters and packet quality parameter thresholds corresponding to each data packet in the at least one video frame, includes: for each frame of the at least one video frame, determining the frame quality parameter whose value is greater than the frame quality parameter threshold as the abnormal frame quality parameter; and for each frame of the at least one video frame, determining the packet quality parameter whose value is greater than the packet quality parameter threshold as the abnormal packet quality parameter.

[0263] In one possible implementation, the streaming media layer protocol information of the data packet includes a frame timestamp indicating the video frame to which the data packet belongs; the network layer protocol information of the data packet includes a packet time indicating the time of the data packet; the frame quality parameters include a frame start packet jitter parameter and a frame end packet jitter parameter, wherein the frame start packet jitter parameter represents the latency jitter of the first data packet corresponding to the video frame, and the frame end packet jitter parameter represents the latency jitter of the last data packet corresponding to the video frame; determining the frame quality parameters of each frame in the at least one video frame based on the streaming media layer protocol information and the network layer protocol information includes: for each frame of the at least one video frame, determining the arrival time of the first packet and the arrival time of the last packet of the video frame based on the packet time of the data packet corresponding to the video frame. The first packet arrival time represents the arrival time of the first data packet corresponding to the video frame, and the last packet arrival time represents the arrival time of the last data packet corresponding to the video frame. Based on the first packet arrival time and the last packet arrival time, the first packet interval and the last packet interval of adjacent video frames are calculated respectively. The first packet interval represents the arrival time difference of the first data packet corresponding to adjacent video frames, and the last packet interval represents the arrival time difference of the last data packet corresponding to adjacent video frames. Based on the frame timestamp, the frame interval of adjacent video frames is calculated respectively. Based on the first packet interval, the frame interval, and the frame first packet jitter parameter of the previous video frame, the frame first packet jitter parameter of the current video frame is calculated. Based on the last packet interval, the frame interval, and the frame last packet jitter parameter of the previous video frame, the frame last packet jitter parameter of the current video frame is calculated.

[0264] In one possible implementation, the streaming media layer protocol information of the data packet includes a streaming media layer sequence number indicating the order of the data packet within its respective video frame; the network layer protocol information of the data packet includes a packet time indicating the time of the data packet; the frame quality parameters include frame loss parameters and frame out-of-order parameters, wherein the frame loss parameters indicate the loss status of data packets in the video frame, and the frame out-of-order parameters indicate the out-of-order status of data packets in the video frame; and the step of determining the frame quality parameters of each frame in the at least one video frame based on the streaming media layer protocol information and the network layer protocol information includes: for each frame of the at least one video frame, determining the frame loss parameters of the video frame based on the streaming media layer sequence number of the data packet corresponding to the video frame; for each frame of the at least one video frame, sorting the data packets corresponding to the video frame according to the chronological order of the packet times of the data packets corresponding to the video frame to obtain a sorting result for the video frame; and determining the frame out-of-order parameters of the video frame based on the sorting result and the streaming media layer sequence number of the data packets corresponding to the video frame.

[0265] In one possible implementation, determining the frame loss parameters for each frame of the at least one video frame based on the streaming media layer sequence number of the data packet corresponding to the video frame includes: for each frame of the at least one video frame, determining the maximum value, minimum value, and number of streaming media layer sequence numbers of the data packet corresponding to the video frame based on the streaming media layer sequence number of the data packet corresponding to the video frame; determining the theoretical value of the number of data packets corresponding to the video frame based on the maximum and minimum values ​​of the streaming media layer sequence numbers; and determining the frame loss parameters for the video frame based on the difference between the theoretical value of the number of data packets corresponding to the video frame and the number of streaming media layer sequence numbers of the data packet corresponding to the video frame. Determining the frame out-of-order parameters for the video frame based on the sorting result and the streaming media layer sequence number of the data packet corresponding to the video frame includes: when the value of the streaming media layer sequence number of the current data packet is greater than the streaming media layer sequence number of the data packets preceding the current data packet in the sorting result, determining the frame out-of-order parameters for the video frame based on the sorting sequence number of the current data packet, the streaming media layer sequence number of the current data packet, and the minimum value of the streaming media layer sequence number of the data packet corresponding to the video frame in the sorting result.

[0266] In one possible implementation, the packet quality parameters of each data packet corresponding to each frame in the at least one video frame are determined based on the network layer protocol information and transport layer protocol information of the data packet. The packet quality parameters include one or more of the following: a packet interval parameter indicating the time interval between the current data packet and the previous data packet; a packet jitter parameter indicating the delay jitter between the current data packet and the previous data packet; an out-of-order parameter indicating the out-of-order status of data packets; a packet loss parameter indicating the packet loss status of data packets; a retransmission parameter indicating the retransmission status of data packets; a delay parameter indicating the delay status of data packets and corresponding response messages; an IP packet fragmentation parameter indicating whether IP packet fragmentation exists in the data packet; and a checksum error parameter indicating whether the checksum of the data packet has an error.

[0267] In one possible implementation, determining the cause and device of the abnormal video quality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters includes: for any video frame, when the frame quality parameters of the video frame and the packet quality parameters of the data packet corresponding to the video frame satisfy a first condition, determining that the cause of the abnormal video quality at the video playback end is a video encoding problem, and determining that the device causing the abnormal video quality at the video playback end is the device used for video encoding; wherein, the first condition includes: the frame packet loss parameters of the video frame are abnormal and the packet loss parameters of the data packet corresponding to the video frame are normal, or the frame out-of-order parameters of the video frame are abnormal and the out-of-order parameters of the data packet corresponding to the video frame are normal.

[0268] In one possible implementation, determining the cause and device of the abnormal video quality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters includes: for any video frame, when the frame quality parameters of the video frame and the packet quality parameters of the data packet corresponding to the video frame satisfy a second condition, determining that the cause of the abnormal video quality at the video playback end is a terminal parameter problem, and determining that the device causing the abnormal video quality at the video playback end is the device used for video playback; wherein, the second condition includes: any one of the frame start packet jitter parameters and frame end packet jitter parameters of the video frame is abnormal, and any one of the IP packet fragmentation parameters and checksum error parameters of the data packet corresponding to the video frame is abnormal.

[0269] In one possible implementation, determining the cause and device for the abnormal video quality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters includes: for any video frame, when the frame quality parameters of the video frame and the packet quality parameters of the data packet corresponding to the video frame satisfy a third condition, determining that the cause of the abnormal video quality at the video playback end is a network transmission problem; wherein, the third condition includes: the frame packet loss parameters of the video frame are abnormal and the packet loss parameters of the data packet corresponding to the video frame are abnormal, or the frame out-of-order parameters of the video frame are abnormal and the out-of-order parameters of the data packet corresponding to the video frame are abnormal, or any one of the frame start packet jitter parameters and frame end packet jitter parameters of the video frame is abnormal and any one of the packet jitter parameters, packet interval parameters, retransmission parameters, and delay parameters of the data packet corresponding to the video frame is abnormal.

[0270] In one possible implementation, after determining that the cause of the abnormal video quality at the video playback end is a network transmission problem, the step of determining the cause and device of the abnormal video quality at the video playback end based on the abnormal frame quality parameters and / or the abnormal packet quality parameters further includes: determining the network layer sequence number of the data packet corresponding to the abnormal packet quality parameters based on the network layer protocol information of the data packet; determining the packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameters based on the transport layer protocol information of the data packet; receiving network topology data from the network device and determining the transmission path of each data packet in the network based on the network topology data; finding data packets on each network device along the transmission path whose packet time, network layer sequence number, transport layer sequence number, and checksum all meet the conditions based on the transmission path of the data packet corresponding to the abnormal packet quality parameters; and determining the network device causing the abnormal video quality at the video playback end from among the network devices along the transmission path of the found data packets with the packet quality parameters that meet the conditions.

[0271] In one possible implementation, the conditions that the packet time, network layer sequence number, transport layer sequence number, and checksum satisfy include: the absolute value of the difference between the packet time and the packet time of the data packet corresponding to the abnormal packet quality parameter is less than a preset threshold, and the network layer sequence number, transport layer sequence number, and checksum are all the same as the network layer sequence number, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameter.

[0272] In one possible implementation, when there are multiple data packets corresponding to the abnormal packet quality parameters, multiple sets of data packets that meet the conditions are found. Based on the packet quality parameters of the found data packets that meet the conditions, the network device causing the abnormal video quality at the video playback end is determined from each network device on the data packet transmission path. This includes: based on the packet quality parameters of multiple sets of data packets that meet the conditions, repeatedly determining the network device that may cause the abnormal video quality at the video playback end from each network device on the data packet transmission path; and statistically analyzing the network device that appears most frequently among the repeatedly determined network devices that may cause the abnormal video quality at the video playback end, and using it as the network device causing the abnormal video quality at the video playback end.

[0273] Embodiments of this application provide a video quality problem determination apparatus, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions.

[0274] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0275] Embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0276] Figure 14 A schematic diagram of an exemplary structure of a video quality problem determination device according to an embodiment of this application is shown.

[0277] like Figure 14 As shown, the video quality problem determination device may include at least one of the following: mobile phone, foldable electronic device, tablet computer, desktop computer, laptop computer, handheld computer, laptop, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device, or smart city device, and server device. This application embodiment does not impose any special limitation on the specific type of the video quality problem determination device.

[0278] The video quality problem determination device may include a processor 110, a memory 121, and a communication module 160. It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the video quality problem determination device. In other embodiments of this application, the video quality problem determination device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0279] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0280] The processor 110 can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.

[0281] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. This memory can store instructions or data that the processor 110 has used or that are used frequently. If the processor 110 needs to use the instruction or data, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0282] The memory 121 can be used to store computer executable program code, including instructions. The memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as an abnormal frame quality parameter determination function, an abnormal packet quality parameter determination function, etc.). The data storage area may store data created during the use of the video quality problem determination device (such as frame quality parameters, packet quality parameters, etc.). Furthermore, the memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional methods of the video quality problem determination device or the aforementioned video quality problem determination method by running instructions stored in the memory 121 and / or instructions stored in memory disposed in the processor.

[0283] The communication module 160 can be used to receive streaming data (e.g., protocol information and network topology data in this embodiment) from other devices or equipment (e.g., the protocol information acquisition device and topology data acquisition device in this embodiment) via wireless / wired communication, and to output streaming data (e.g., information about the causes of video quality problems and devices determined in this embodiment) to other devices or equipment. For example, it can provide wireless communication solutions including WLAN (such as Wi-Fi networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR).

[0284] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0285] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0286] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0287] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0288] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0289] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0290] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0291] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0292] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0293] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of video quality problem determination, the method comprising: The method comprises: receiving a data packet of a video stream type on a video player and a network device, the data packet comprising a payload header and a payload, the payload header comprising protocol information, the network device being configured to transmit video data to the video player, the protocol information indicating a sending mode of a video frame, the video frame comprising a plurality of data packets; determining an abnormal frame quality parameter and an abnormal packet quality parameter according to the protocol information, the abnormal frame quality parameter indicating a quality abnormality of a video frame to which the abnormal frame quality parameter belongs, the abnormal packet quality parameter indicating a quality abnormality of a data packet to which the abnormal packet quality parameter belongs; determining a cause of a video quality abnormality of the video player and a device according to the abnormal frame quality parameter and / or the abnormal packet quality parameter.

2. The method of claim 1, wherein, The protocol information comprises streaming layer protocol information, network layer protocol information and transmission layer protocol information of the data packet, the streaming layer protocol information comprising streaming layer protocol information of the data packet on the video player, the network layer protocol information comprising network layer protocol information of the data packet on the network device and on the video player, the transmission layer protocol information comprising transmission layer protocol information of the data packet on the network device and on the video player, The determining of the abnormal frame quality parameter and the abnormal packet quality parameter according to the protocol information comprises: determining at least one video frame and respective data packets corresponding to respective frames in the at least one video frame according to the streaming layer protocol information; determining frame quality parameters of respective frames in the at least one video frame according to the streaming layer protocol information and the network layer protocol information; determining packet quality parameters of respective data packets corresponding to respective frames in the at least one video frame according to the network layer protocol information and the transmission layer protocol information; determining the abnormal frame quality parameter according to respective frame quality parameters corresponding to respective frames in the at least one video frame and a frame quality parameter threshold value, and determining the abnormal packet quality parameter according to respective packet quality parameters of respective data packets corresponding to respective frames in the at least one video frame and a packet quality parameter threshold value.

3. The method of claim 2, wherein, The determining of the abnormal frame quality parameter according to respective frame quality parameters corresponding to respective frames in the at least one video frame and a frame quality parameter threshold value, and the determining of the abnormal packet quality parameter according to respective packet quality parameters of respective data packets corresponding to respective frames in the at least one video frame and a packet quality parameter threshold value, comprises: for each frame in the at least one video frame, determining, from frame quality parameters of the video frame, a frame quality parameter having a value greater than the frame quality parameter threshold value as the abnormal frame quality parameter; for each frame in the at least one video frame, determining, from packet quality parameters of data packets corresponding to the video frame, a packet quality parameter having a value greater than the packet quality parameter threshold value as the abnormal packet quality parameter.

4. The method according to claim 2 or 3, characterized in that, The stream media layer protocol information of the data packet comprises a frame timestamp indicating a video frame to which the data packet belongs, the network layer protocol information of the data packet comprises a packet time indicating a time of the data packet, and the frame quality parameter comprises a frame head packet jitter parameter and a frame tail packet jitter parameter, the frame head packet jitter parameter representing a time delay jitter condition of a first data packet corresponding to a video frame, and the frame tail packet jitter parameter representing a time delay jitter condition of a last data packet corresponding to the video frame, The determining of the frame quality parameter of each frame in the at least one video frame according to the stream media layer protocol information and the network layer protocol information comprises: For each frame of the at least one video frame, a head packet arrival time and a tail packet arrival time of the video frame are determined according to a packet time of a data packet corresponding to the video frame, the head packet arrival time representing an arrival time of a first data packet corresponding to the video frame, and the tail packet arrival time representing an arrival time of a last data packet corresponding to the video frame; A head packet interval and a tail packet interval of adjacent video frames are respectively calculated according to the head packet arrival time and the tail packet arrival time, the head packet interval representing an arrival time difference of first data packets corresponding to adjacent video frames, and the tail packet interval representing an arrival time difference of last data packets corresponding to the adjacent video frames; A frame interval of the adjacent video frames is calculated according to the frame timestamp; The frame head packet jitter parameter of the current video frame is calculated according to the head packet interval, the frame interval, and a frame head packet jitter parameter of a previous video frame; The frame tail packet jitter parameter of the current video frame is calculated according to the tail packet interval, the frame interval, and a frame tail packet jitter parameter of the previous video frame.

5. The method according to claim 2 or 3, characterized in that, The stream media layer protocol information of the data packet comprises a stream media layer sequence number indicating an order of the data packet in a video frame to which the data packet belongs, the network layer protocol information of the data packet comprises a packet time indicating a time of the data packet, and the frame quality parameter comprises a frame packet loss parameter and a frame out-of-order parameter, the frame packet loss parameter representing a loss condition of data packets of a video frame, and the frame out-of-order parameter representing an out-of-order condition of the data packets of the video frame, The determining of the frame quality parameter of each frame in the at least one video frame according to the stream media layer protocol information and the network layer protocol information comprises: For each frame of the at least one video frame, a frame packet loss parameter of the video frame is determined according to a stream media layer sequence number of a data packet corresponding to the video frame; For each frame of the at least one video frame, the data packets corresponding to the video frame are sorted according to a sequence of packet times of the data packets, to obtain a sorting result of the data packets corresponding to the video frame, and a frame out-of-order parameter of the video frame is determined according to the sorting result and the stream media layer sequence number of the data packets corresponding to the video frame.

6. The method of claim 5, wherein, The determining of the frame packet loss parameter of each frame of the at least one video frame according to the stream media layer sequence number of the data packet corresponding to the video frame comprises: For each frame of the at least one video frame, according to the stream media layer sequence number of the data packet corresponding to the video frame, a maximum value, a minimum value and a number of stream media layer sequence numbers of the data packet corresponding to the video frame are determined, according to the maximum value and the minimum value of the stream media layer sequence number, a theoretical value of the number of data packets corresponding to the video frame is determined, and according to the difference between the theoretical value of the number of data packets corresponding to the video frame and the number of stream media layer sequence numbers of the data packet corresponding to the video frame, a frame packet loss parameter of the video frame is determined; According to the sorting result and the stream media layer sequence number of the data packet corresponding to the video frame, a frame disorder parameter of the video frame is determined, including: When the numerical value of the stream media layer sequence number of the current data packet is greater than the stream media layer sequence number of the data packet before the current data packet in the sorting result, according to the sorting sequence number of the current data packet in the sorting result, the stream media layer sequence number of the current data packet, and the minimum value of the stream media layer sequence number of the data packet corresponding to the video frame, a frame disorder parameter corresponding to the video frame is determined.

7. The method of claim 2 or 3, wherein, The packet quality parameters of the data packets corresponding to each frame in the at least one video frame are determined according to the network layer protocol information and the transport layer protocol information of the data packets, and the packet quality parameters include one or more of a packet interval parameter representing the time interval between the current data packet and the previous data packet, a packet jitter parameter representing the time delay jitter between the current data packet and the previous data packet, a disorder parameter representing the data packet disorder situation, a packet loss parameter representing the data packet loss situation, a retransmission parameter representing the data packet retransmission situation, a time delay parameter representing the time delay situation of the data packet and the corresponding response message, an IP message fragmentation parameter representing whether the data packet exists IP message fragmentation, and a checksum error parameter representing whether the checksum of the data packet is incorrect.

8. The method of claim 7, wherein, The determination of the cause and the device causing the video quality abnormality of the video playback end according to the abnormal frame quality parameter and / or the abnormal packet quality parameter includes: When the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame satisfy a first condition, it is determined that the cause of the video quality abnormality of the video playback end is a video encoding problem, and the device causing the video quality abnormality of the video playback end is the device used for video encoding. The first condition includes: The frame packet loss parameter of the video frame is abnormal and the packet loss parameter of the data packet corresponding to the video frame is normal, or The frame disorder parameter of the video frame is abnormal and the disorder parameter of the data packet corresponding to the video frame is normal.

9. The method of claim 7, wherein, The determination of the cause and the device causing the video quality abnormality of the video playback end according to the abnormal frame quality parameter and / or the abnormal packet quality parameter includes: When the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame satisfy a second condition, it is determined that the cause of the video quality abnormality of the video playback end is a terminal parameter problem, and the device causing the video quality abnormality of the video playback end is the device used for video playback. The second condition includes: Any one of the frame head packet jitter parameter and the frame tail packet jitter parameter of the video frame is abnormal, and any one of the IP message fragmentation parameter, the checksum error parameter of the data packet corresponding to the video frame is abnormal.

10. The method of claim 7, wherein, The method further comprises: When the frame quality parameter of any video frame and the packet quality parameter of the data packet corresponding to the video frame satisfy a third condition, determining that the reason for the video quality abnormality of the video playback end is a network transmission problem. The third condition comprises: The frame loss packet parameter of the video frame is abnormal, and the loss packet parameter of the data packet corresponding to the video frame is abnormal, or The frame out-of-order parameter of the video frame is abnormal, and the out-of-order parameter of the data packet corresponding to the video frame is abnormal, or Any one of the frame head packet jitter parameter and the frame tail packet jitter parameter of the video frame is abnormal, and any one of the packet jitter parameter, the packet interval parameter, the retransmission parameter, the time delay parameter of the data packet corresponding to the video frame is abnormal.

11. The method of claim 10, wherein, After determining that the reason for the video quality abnormality of the video playback end is a network transmission problem, the method further comprises: determining the network layer sequence number of the data packet corresponding to the abnormal packet quality parameter according to the network layer protocol information of the data packet; determining the packet time, the transport layer sequence number and the checksum of the data packet corresponding to the abnormal packet quality parameter according to the transport layer protocol information of the data packet; receiving network topology data on the network device, and determining the transmission path of each data packet in the network according to the network topology data; finding the data packet that satisfies the condition in each network device on the transmission path of the data packet corresponding to the abnormal packet quality parameter according to the transmission path of the data packet corresponding to the abnormal packet quality parameter in the network; determining the network device causing the video quality abnormality of the video playback end from each network device on the transmission path of the data packet according to the packet quality parameter of the found data packet that satisfies the condition.

12. A video quality problem determination apparatus characterized by comprising: The apparatus comprises: an information receiving module configured to receive a data packet of a video stream type on a video playback end and a network device, the data packet comprising a payload header and a payload, the payload header comprising protocol information, the network device being configured to transmit video data to the video playback end, the protocol information indicating a sending mode of a video frame, the video frame comprising a plurality of data packets; a parameter determining module configured to determine an abnormal frame quality parameter and an abnormal packet quality parameter according to the protocol information, the abnormal frame quality parameter indicating quality abnormality of a video frame to which the abnormal frame quality parameter belongs, and the abnormal packet quality parameter indicating quality abnormality of a data packet to which the abnormal packet quality parameter belongs; an abnormal reason determining module configured to determine a reason for video quality abnormality of the video playback end and a device according to the abnormal frame quality parameter and / or the abnormal packet quality parameter.

13. A video quality problem determination apparatus characterized by comprising: comprise: a processor; a memory for storing processor-executable instructions; The processor is configured to implement the method of any one of claims 1-11 when executing the instructions.

14. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of any one of claims 1-11.

15. A computer program product comprising computer readable code, or a non-transitory computer readable storage medium having computer readable code embodied thereon, the computer readable code comprising: code for receiving a request for a service from a user equipment (UE) in a wireless communication system; code for determining whether the UE is authorized to use the service; and code for providing the service to the UE if the UE is authorized to use the service. When the computer readable code is run in an electronic device, a processor in the electronic device implements the method of any one of claims 1-11.

Images