Method for improving playing quality of video data, video streaming media system and equipment

Abstract

The embodiment of the invention provides a method for improving the playing quality of video data, a video streaming media system and equipment, relates to the field of articulated naturality web, and is applied to the video streaming media system, the system comprises a video sending end, a cloud end and a video playing end, and the method comprises the steps that the cloud end receives first video data sent by the video sending end and carries out error frame detection, when an error frame is detected, repairing is carried out based on the piece data and the macro block data of the error frame, second video data is determined, and after a video acquisition request of a video playing end is received, the second video data is sent to the video playing end; the video playing end carries out frame rate detection and frame missing detection on the received second video data, when continuous frame missing is detected, continuous frame compensation is carried out on the second video data, and third video data is determined and played. The cloud end and the video playing end carry out cooperative processing, and the video restoration quality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video live streaming, and in particular to a method for improving video data playing quality, a video streaming system and a device. BACKGROUND

[0002] With the development of network and multimedia technologies, cloud-based services that require both live and recorded video storage, such as video monitoring, live streaming, and online meetings, are accelerating. Video streams are transmitted from a video sending end to a cloud end through various protocols, and after protocol conversion, storage / enhancement processing, and distribution to a video playing end through a playing protocol suitable for different scenarios, end-to-end audio and video services are achieved.

[0003] During video stream transmission, factors such as network congestion and unstable links often cause packet loss or damage during video transmission, resulting in problems such as freezing, screen tearing, and even playback interruption. In the prior art, when packet loss occurs, the user's viewing experience is improved by analyzing the received video data at the cloud end, determining the missing macroblock video frame, and discarding it or re-uploading the video data at the video sending end.

[0004] However, discarding the missing macroblock video frame can cause freezing and frame skipping; when the network is severely congested, even retransmission can still cause video playback to freeze and fail to recover smoothly. SUMMARY

[0005] The embodiments of the present application provide a method for improving video data playing quality, a video streaming system, and a device to implement macroblock-level video data repair, avoid screen tearing, improve video repair quality, and enhance user playback experience.

[0006] In a first aspect, the embodiments of the present application provide a method for improving video data playing quality, applied to a cloud end of a video streaming system, wherein the video streaming system further includes a video sending end, the cloud end receives first video data sent by the video sending end, and performs error frame detection based on the first video data, and the method includes:

[0007] When an error frame is detected, the error frame is repaired based on the slice data and macroblock data of the error frame to obtain a repaired frame;

[0008] Based on the repaired frame, second video data is determined, which is sent to a video playing end after receiving a video acquisition request from the video playing end.

[0009] In one possible implementation, the error frame is repaired based on the slice data and macroblock data of the error frame to obtain a repaired frame, including:

[0010] obtain network abstraction layer data of the error frame, determine a frame type corresponding to the error frame based on the network abstraction layer data;

[0011] analyze the network abstraction layer data to obtain slice data of the error frame, and determine whether the error frame can be subjected to macroblock error correction according to a preset condition, the preset condition including that a slice data abnormality degree exceeds a first preset threshold and that the slice data does not conform to a first syntax specification;

[0012] If not, perform single-frame compensation on the error frame to obtain a repaired frame;

[0013] If yes, perform macroblock checking on a macroblock in the error frame based on the frame type and the macroblock data, determine an abnormal macroblock corresponding to the error frame, and perform macroblock error correction on the abnormal macroblock, and determine a repaired frame based on the abnormal macroblock after the error correction.

[0014] In a possible implementation, the frame type includes a first type, the macroblock data includes macroblock residual data and a motion vector, and the abnormal macroblock includes any one of a first abnormal macroblock whose macroblock data does not conform to a second syntax specification, a second abnormal macroblock whose macroblock residual data exceeds a second preset threshold, and a third abnormal macroblock whose motion vector exceeds a third preset threshold.

[0015] In a possible implementation, the macroblock error correction on the abnormal macroblock includes:

[0016] When the frame type is the first type, determine a normal macroblock of the abnormal macroblock in a preset area, and determine effective data corresponding to a motion vector of the normal macroblock and a number of the effective data;

[0017] Determine whether the number of the effective data is less than a fourth preset threshold;

[0018] If yes, use a motion vector of a first associated macroblock corresponding to a reference frame of the abnormal macroblock as the motion vector of the abnormal macroblock, the reference frame being determined based on a front reference frame index in the macroblock data of the abnormal macroblock;

[0019] If not, repair the motion vector of the abnormal macroblock based on the effective motion vector of the normal macroblock, and determine a second associated macroblock corresponding to a reference frame of the abnormal macroblock after the repair;

[0020] Determine a search range based on the second associated macroblock, calculate a matching degree between a candidate macroblock in the search range and the abnormal macroblock, and determine a matching macroblock based on the matching degree, and replace the abnormal macroblock with the matching macroblock.

[0021] In a possible implementation, the frame type further includes a second type, and the macroblock error correction on the abnormal macroblock includes:

[0022] When the frame type is the second type, the neighboring macroblock of the abnormal macroblock is extracted;

[0023] Edge detection is performed on the neighboring macroblock to determine a weight corresponding to the neighboring macroblock;

[0024] A second pixel value is determined based on the weight and a first pixel value corresponding to the neighboring macroblock, and interpolation processing is performed on the abnormal macroblock based on the second pixel value;

[0025] The abnormal macroblock after the interpolation processing is subjected to texture completion optimization to repair an area texture missing region.

[0026] In a possible implementation, the error frame is subjected to single-frame compensation to obtain a repaired frame, and the method includes:

[0027] A previous frame and a subsequent frame of the error frame are determined, the previous frame and the subsequent frame are input into an optical flow model to obtain a forward optical flow corresponding to the previous frame and a backward optical flow corresponding to the subsequent frame, and the optical flow model is obtained based on a sample frame;

[0028] Each pixel of the error frame is subjected to motion compensation according to the forward optical flow and the backward optical flow to determine a corresponding mapping pixel value;

[0029] A repaired frame is determined based on the mapping pixel value of each pixel.

[0030] In a possible implementation, the video streaming system further includes a video playback end, the video playback end includes a front-end player and a native back-end, and before the error frame is subjected to repair processing based on the slice data and the macroblock data of the error frame to obtain a repaired frame, the method further includes:

[0031] Mode information sent by the video playback end is received, the mode information is configured by a user on the front-end player, and includes a first playback mode and a second playback mode;

[0032] When the mode information is the first playback mode, the received first video data is subjected to repair processing, and after a video acquisition request sent by the native back-end is received, the second video data is sent to the native back-end;

[0033] When the mode information is the second playback mode, after a video acquisition request sent by the video playback end is received, fourth video data is sent to the front-end player, and the fourth video data is obtained by encapsulating the first video data after the analysis processing.

[0034] In a second aspect, the embodiments of the present application provide a method for improving the playing quality of video data, applied to a video playing end of a video streaming media system, and the method comprises the following steps:

[0035] receiving second video data sent by the cloud, performing frame rate detection and missing frame detection on the second video data, wherein the second video data is obtained by repairing the received first video data by the cloud according to the first aspect and / or various possible implementations of the first aspect;

[0036] when the number of missing video frames of the second video data reaches a fifth preset threshold, performing continuous frame compensation on the second video data, obtaining third video data based on the second video data after the continuous frame compensation, and playing the third video data.

[0037] In a possible implementation, the continuous frame compensation on the second video data comprises:

[0038] determining a group of images based on the second video data, extracting spatio-temporal features in video frames in the group of images through a convolution layer of a diffusion model, wherein the diffusion model is trained based on sample data;

[0039] capturing cross-frame dependency of the spatio-temporal features through a time sequence attention layer, and establishing a motion model of objects in the group of images;

[0040] generating continuous frames through the diffusion model and the spatio-temporal features, the cross-frame dependency, and the motion model.

[0041] In a possible implementation, the video playing end comprises a native back end and a front-end player; before the receiving of the second video data sent by the cloud and the frame rate detection and the missing frame detection on the second video data, the method further comprises:

[0042] the front-end player determines a connection object according to mode information configured by a user, wherein the mode information comprises a first playing mode and a second playing mode;

[0043] wherein, when the mode information is the first playing mode, the connection object is determined as the native back end, and third video data sent by the native back end is received and played;

[0044] when the mode information is the second playing mode, the connection object is determined as the cloud, and fourth video data sent by the cloud is received and played, wherein the fourth video data is obtained by encapsulating the first video data after the parsing processing.

[0045] In a third aspect, the embodiments of the present application provide a video streaming media system, comprising a video sending end and a cloud, wherein the video sending end is in communication connection with the cloud.

[0046] The video sending end is configured to send the first video data to the cloud end.

[0047] The cloud end is configured to receive the first video data sent by the video sending end, and perform error frame detection based on the first video data. When an error frame is detected, the received first video data is repaired by using the first aspect and / or various possible implementation manners of the first aspect.

[0048] In a possible implementation manner, the frequency stream media system further comprises a video playing end in communication connection with the cloud end. The video playing end performs continuous frame compensation on the received second video data by using the first aspect and / or various possible implementation manners of the first aspect.

[0049] In a possible implementation manner, the video playing end comprises a native back end and a front-end player.

[0050] The native back end performs continuous frame compensation on the second video data, obtains third video data based on the second video data after the continuous frame compensation, and sends the third video data to the front-end player.

[0051] The front-end player receives and plays the third video data.

[0052] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor.

[0053] The memory stores computer execution instructions.

[0054] The processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect, and executes the first aspect and / or various possible implementation manners of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer execution instructions. The computer execution instructions are executed by a processor to realize the first aspect and / or various possible implementation manners of the first aspect, and to execute the first aspect and / or various possible implementation manners of the first aspect.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises a computer program. The computer program is executed by a processor to realize the first aspect and / or various possible implementation manners of the first aspect, and to execute the first aspect and / or various possible implementation manners of the first aspect.

[0057] The method for improving the video data playing quality, the video streaming media system and the device provided by the embodiments of the present application realize macroblock-level video data repair by performing repair processing on the error frame based on the slice data and the macroblock data of the error frame when the error frame is detected, avoid the screen flashing phenomenon of the video, improve the quality of video repair, and improve the user playing experience by determining the second video data sent to the video playing end after the video acquisition request is received according to the repair frame. BRIEF DESCRIPTION OF DRAWINGS

[0058] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 The scene schematic diagram of the video streaming in the prior art provided by the embodiments of the present application;

[0060] Figure 2 The flowchart of the method for improving the video data playing quality provided by the embodiments of the present application Figure One ;

[0061] Figure 3 The flowchart of the method for improving the video data playing quality provided by the embodiments of the present application Figure Two ;

[0062] Figure 4 The flowchart of the method for improving the video data playing quality provided by the embodiments of the present application Figure Three ;

[0063] Figure 5 The structure schematic diagram of the video streaming media system provided by the embodiments of the present application Figure One ;

[0064] Figure 6 The structure schematic diagram of the video streaming media system provided by the embodiments of the present application Figure Two ;

[0065] Figure 7 The structure schematic diagram of the video streaming media system provided by the embodiments of the present application Figure Three ;

[0066] Figure 8 The structure schematic diagram of the electronic device provided by the embodiments of the present application.

[0067] The specific embodiments of the present application have been shown in the above drawings, and will be described in more detail hereinafter. The drawings and the written description are not intended to limit the scope of the present application in any way, but to explain the concept of the present application to those skilled in the art by referring to the specific embodiments. DETAILED DESCRIPTION

[0068] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description herein refers to the accompanying drawings, which show by way of illustration various

[0069] Firstly, the terms involved in the present application are explained:

[0070] Video: A video is composed of a plurality of video sequences, a video sequence is composed of a plurality of frames, a frame is composed of a plurality of slices, a slice is composed of a plurality of macroblocks, and a macroblock is composed of a plurality of blocks.

[0071] With the development of network and multimedia technologies, cloud-based services that require simultaneous live and recorded video storage, such as video monitoring, live streaming, and network conferencing, are accelerating.

[0072] Figure 1 The prior art provides a scene diagram for video stream transmission for the embodiments of the present application, as shown in Figure 1 The video sending end 101 transmits protocol video data to the cloud end 102 according to various protocols, the cloud end 102 performs protocol conversion, storage / enhancement, and other processing on the received video data, and then distributes the playback video data to the video playback end 103 through a playback protocol adapted to different scenarios. During the video data transmission process, factors such as network congestion and unstable links often cause packet loss or damage during video transmission, resulting in problems such as freezing, screen tearing, and even playback interruption. In the prior art, when packet loss occurs, the user's viewing experience is improved by analyzing the received video data at the cloud end 102 to determine the video frames that lack macroblocks and discard them, or by re-uploading the video data from the video sending end 101.

[0073] In combination with the above scenario, it can be seen that in the prior art, discarding video frames that lack macroblocks can cause freezing and frame skipping. In severe network congestion, even retransmission can still cause freezing and screen tearing during video playback, and cannot restore smooth playback in time.

[0074] The method for improving video data playback quality provided by the present application detects error frames at the cloud end, repairs the error frames using slice data and macroblock data of the error frames, and solves the problem of freezing and screen tearing in the prior art when video data is lost.

[0075] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes can not be described again in some examples. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0076] Figure 2 Flowchart of a method for improving video data playing quality provided by an embodiment of the present application Figure One , applied to a cloud end of a video streaming system, the video streaming system further comprising a video sending end; the cloud end receives first video data sent by the video sending end and performs error frame detection based on the first video data, such as Figure 2 indicated, the method comprises:

[0077] S201, when an error frame is detected, performing repair processing on the error frame based on slice data and macroblock data of the error frame to obtain a repaired frame;

[0078] When performing error frame detection based on the first video data, the first video can be parsed according to a video transmission protocol to determine the error frame through protocol disassembly; a machine learning method can also be used to input the received first video data into a trained error frame detection model, and the error frame detection model judges whether the video frame is an error frame according to the learned feature pattern. The error frame detection model is obtained by training a large number of video sample data containing error frames and correct frames and corresponding labels collected in the cloud end.

[0079] A number of blocks constitute a macroblock, a number of slices constitute a frame, a number of frames constitute a video sequence, and a number of video sequences constitute a video. Video decoding is finally simplified to search and retrieve macroblocks from data streams, and pixel colors are restored by means of luminance and chrominance components. Therefore, the video decoding produces a screen with a large number of blocks, which is caused by insufficient or incorrect macroblocks after data analysis of a frame.

[0080] When an error frame is detected, the error frame is repaired based on error points in the slice data and the macroblock data of the error frame to avoid insufficient or incorrect macroblocks, and a repaired frame is determined based on the error frame after the repair processing.

[0081] S202, determining second video data based on the repaired frame.

[0082] The second video data is used to send to a video playing end after receiving a video acquisition request of the video playing end.

[0083] Based on receiving a video acquisition request of a video playing end, a playing protocol corresponding to the video acquisition request is determined, and second video data is determined based on the repaired frame and the playing protocol.

[0084] In a possible implementation, the video streaming system further comprises a video playback end, the video playback end comprising a front-end player and a native back-end, before the error frame is repaired based on the slice data and the macroblock data of the error frame to obtain a repaired frame, the method further comprises:

[0085] receiving mode information sent by the video playback end, the mode information comprising a first playback mode and a second playback mode; when the mode information is the first playback mode, performing repair processing on the received first video data, and after receiving a video acquisition request sent by the native back-end, sending the second video data to the native back-end; when the mode information is the second playback mode, after receiving a video acquisition request sent by the video player, sending the fourth video data to the front-end player.

[0086] The mode information is configured by a user on the front-end player, and the fourth video data is obtained by encapsulating the first video data after the parsing processing.

[0087] The mode information configured on the video playback end is used to determine whether to perform macroblock-level error correction, when the mode information is the first playback mode, error frame determination and macroblock-level repair are performed, the video playback quality is improved, when the mode information is the second playback mode, after receiving a video acquisition request sent by the video player, the fourth video data is sent to the front-end player, the network degradation condition is found more quickly through the original data condition, and the adaptability of the video streaming system is improved.

[0088] The method for improving video data playback quality provided in the embodiments of the application implements macroblock-level video data repair by performing repair processing on the error frame based on the slice data and the macroblock data of the error frame when the error frame is detected, avoids the screen flashing phenomenon of the video, improves the video repair quality, and determines the second video data sent to the video playback end after receiving a video acquisition request according to the repaired frame, thereby improving the playback experience of the user.

[0089] Figure 3 The method for improving video data playback quality provided in the embodiments of the application is shown in the flowchart Figure Two The embodiments of the application are based on Figure 2 The method for improving video data playback quality provided in the embodiments of the application is shown in the flowchart

[0090] S301, network abstract layer data of an error frame is obtained, and a frame type corresponding to the error frame is determined based on the network abstract layer data;

[0091] The Network Abstraction Layer Unit (NAL data) refers to a basic transmission unit of video coding.

[0092] The NAL data of the error frame is obtained, and the frame type corresponding to the error frame is determined according to the byte information of the NAL data. The frame type may be, for example, an I frame, a P frame, or the like.

[0093] S302, the network abstraction layer data is parsed to obtain the slice data of the error frame, and it is judged whether the error frame can be macroblock error corrected according to a preset condition; if not, step S303 is executed; if yes, step S304 is executed.

[0094] The preset condition includes that the abnormality degree exceeds a first preset threshold and the slice data does not conform to a first syntax specification; the slice data refers to the encoding data corresponding to the slice.

[0095] The NAL data is parsed to obtain the slice data corresponding to the error frame. When the abnormality degree exceeds the first preset threshold, it indicates that there is too much error data or missing data in the slice data, and the macroblock error correction cannot be performed. When the slice data does not conform to the first syntax specification, the macroblock-level error correction also cannot be performed. Therefore, for the video frame that cannot be macroblock error corrected, single-frame compensation is performed to obtain a repaired frame. For the video frame that can be macroblock error corrected, the macroblock in the error frame is checked based on the frame type and the macroblock data, the abnormal macroblock corresponding to the error frame is determined, and the macroblock error correction is performed on the abnormal macroblock. The repaired frame is determined based on the error-corrected abnormal macroblock.

[0096] S303, single-frame compensation is performed on the error frame to obtain a repaired frame.

[0097] In a possible implementation, the single-frame compensation on the error frame to obtain the repaired frame is described in detail, including:

[0098] The previous frame and the next frame of the error frame are determined, the previous frame and the next frame are input into an optical flow model to obtain a forward optical flow corresponding to the previous frame and a backward optical flow corresponding to the next frame, motion compensation is performed on each pixel of the error frame according to the forward optical flow and the backward optical flow to determine a corresponding mapped pixel value, and the repaired frame is determined based on the mapped pixel value of each pixel.

[0099] The optical flow model is trained based on sample frames. The previous frame refers to a previous video frame of the error frame. The previous frame may be, for example, a previous frame of the error frame. The next frame refers to a next video frame of the video frame. The next frame may be, for example, a next frame of the error frame. The forward optical flow refers to a pixel motion vector from the previous frame to the error frame. The backward optical flow refers to a pixel motion vector from the next frame to the error frame.

[0100] The front frame and the rear frame are determined according to the error frame, and the front frame and the rear frame are input into an optical flow model to calculate a forward optical flow corresponding to the front frame and a backward optical flow corresponding to the rear frame respectively,

[0101] The following formula is used to perform motion compensation on each pixel of the error frame to determine a corresponding mapping pixel value:

[0102]

[0103] wherein, is the mapping pixel value of the pixel, is an optical flow confidence corresponding to the forward optical flow; is a forward optical flow offset, is the forward optical flow; is an optical flow confidence corresponding to the backward optical flow; is a backward optical flow offset; is the backward optical flow.

[0104] The mapping pixel value of the pixel after motion compensation is used to determine the repair frame.

[0105] S304, based on the frame type and the macroblock data, performing macroblock checking on the macroblocks in the error frame to determine abnormal macroblocks corresponding to the error frame, and performing macroblock error correction on the abnormal macroblocks, and based on the corrected abnormal macroblocks, determining the repair frame.

[0106] Through layer-by-layer data analysis, multi-level video data packet loss repair processing is realized, and the video repair quality is improved.

[0107] In a possible implementation, the frame type includes a first type, and the macroblock data includes macroblock residual data and a motion vector. The abnormal macroblock is described in detail as follows.

[0108] Any one of a first abnormal macroblock in which the macroblock data does not conform to a second syntax specification, a second abnormal macroblock in which the macroblock residual data exceeds a second preset threshold, and a third abnormal macroblock in which the motion vector exceeds a third preset threshold.

[0109] The first type refers to a video frame dependent on a previous frame, for example, a P frame.

[0110] For each macroblock, it is determined whether the macroblock data of the macroblock conforms to the second syntax specification. When the macroblock data does not conform to the second syntax specification, the macroblock is determined to be the first abnormal macroblock. If the macroblock data conforms to the second syntax specification, it is determined whether the macroblock residual data of the macroblock exceeds the second preset threshold. When the macroblock residual data exceeds the first preset threshold, the macroblock is determined to be the second abnormal macroblock.

[0111] Alternatively, when the frame type is the first type, it is determined whether the motion vector of the macroblock exceeds a third preset threshold value, and when the motion vector exceeds the third preset threshold value, the macroblock is determined as the third abnormal macroblock.

[0112] In a possible implementation, the macroblock error correction on the abnormal macroblock is described in detail, including:

[0113] When the frame type is the first type, it is determined whether the motion vector of the macroblock exceeds a third preset threshold value, and when the motion vector exceeds the third preset threshold value, the macroblock is determined as the third abnormal macroblock.

[0114] The reference frame is determined based on a front reference frame index in the macroblock data of the abnormal macroblock.

[0115] The preset area may be, for example, a 3*3 area, and the fourth preset threshold value is related to the size of the preset area, and may be, for example, 3, 4, etc.

[0116] The abnormal macroblock in the preset area is determined as a normal macroblock, and the invalid motion vector in the normal macroblock is removed, for example, by a median filtering method to remove the invalid motion vector, to determine the effective data in the preset area and the corresponding effective data quantity. It is determined whether the effective data quantity is less than the fourth preset threshold value. If the effective data quantity is less than the preset threshold value, it indicates that the reliability of the macroblock error correction based on the motion vector of the normal macroblock in the preset area is not high. Therefore, the motion vector of the macroblock at the corresponding position of the reference frame of the abnormal macroblock, i.e., the motion vector of the first associated macroblock, is used as the motion vector of the abnormal macroblock.

[0117] If not, the motion vector of the abnormal macroblock is repaired based on the effective motion vector of the normal macroblock, and a second associated macroblock corresponding to the repaired abnormal macroblock in the reference frame is determined, a search range is determined based on the second associated macroblock, a matching degree between a candidate macroblock in the search range and the abnormal macroblock is calculated, and a matching macroblock is determined based on the matching degree, and the abnormal macroblock is replaced by the matching macroblock.

[0118] The matching degree may be, for example, measured by the sum of absolute differences between the abnormal macroblock and the candidate macroblock, or by the sum of square differences between the abnormal macroblock and the candidate macroblock.

[0119] Based on the valid motion vector of the normal macroblock, the motion vector of the abnormal macroblock is repaired, and the second associated macroblock corresponding to the repaired abnormal macroblock in the reference frame is determined. With the second associated macroblock as the center, a preset pixel range is determined as the search range. The preset pixel range can be, for example, 32 pixels. The degree of match between the candidate macroblocks and the abnormal macroblock is determined based on the blocks in the candidate macroblock within the search range.

[0120] When determining a matching macroblock by the sum of absolute differences, the following formula is used to calculate the sum of absolute differences between the candidate macroblock and the abnormal macroblock:

[0121]

[0122] in, is the sum of absolute differences, For candidate macroblocks Pixel value of For abnormal macroblocks pixel value; N can be determined according to the blocks in the macroblock, or it can be preset, for example, it can be 16.

[0123] Based on the matching degree, the macroblock that best matches the abnormal macroblock is determined as the matching macroblock, and the abnormal macroblock is replaced with the matching macroblock. For example: When measuring the matching degree, The smallest candidate macroblock is the matching macroblock.

[0124] In a possible implementation, the frame type further includes a second type, and the macroblock error correction for the abnormal macroblock is described in detail, including:

[0125] When the frame type is the second type, adjacent macroblocks of the abnormal macroblock are extracted; edge detection is performed on the adjacent macroblocks to determine the weights corresponding to the adjacent macroblocks; a second pixel value is determined based on the weight and the first pixel value corresponding to the adjacent macroblock, and interpolation processing is performed on the abnormal macroblock based on the second pixel value; texture completion optimization is performed on the abnormal macroblock after the interpolation processing to repair the area with missing texture.

[0126] The second type refers to an independent video frame, such as an I-frame. Adjacent macroblocks refer to adjacent macroblocks.

[0127] When the frame type is the second type, the adjacent macroblocks of the abnormal macroblock are extracted, for example, the adjacent normal macroblocks above, below, left, and right are detected. The edge of the adjacent macroblocks is monitored to determine the edge direction and edge strength to determine the weight corresponding to the adjacent macroblock.

[0128] The second pixel value is calculated using the following formula:

[0129]

[0130] wherein, is a second pixel value, is a weight corresponding to the adjacent macroblock; is a first pixel value of the adjacent macroblock, and n is a number of normal macroblocks, for example, can be 4.

[0131] Based on the second pixel value, the abnormal macroblock is interpolated; and the abnormal macroblock after the interpolation is optimized for texture completion to repair the area texture missing region.

[0132] The method for improving the video data playing quality provided by the embodiment of the application acquires the network abstraction layer data of the error frame, determines the frame type corresponding to the error frame based on the network abstraction layer data, and performs the encoding layer data checking after the protocol layer analysis to the error frame, so that the calculation complexity is reduced layer by layer; the network abstraction layer data is analyzed to obtain the slice data of the error frame, the abnormality degree of the slice data and the syntax of the slice data are judged to determine whether the slice data of the error frame can be corrected by the macroblock, the single frame compensation is performed to the error frame when the slice data cannot be corrected by the macroblock to obtain the repair frame, and the frame integrity is ensured; when the slice data can be corrected by the macroblock, the macroblock in the error frame is checked based on the frame type and the macroblock data, the abnormal macroblock corresponding to the error frame is determined, and the macroblock correction is performed to the abnormal macroblock, the repair frame is determined based on the corrected abnormal macroblock, the macroblock level video data repair is realized, the video screen flashing phenomenon is avoided, the video repair quality is improved, and the user playing experience is improved.

[0133] During the video playing, in addition to the screen flashing phenomenon affecting the user playing experience, the multi-frame missing leads to the video freezing phenomenon, which also affects the user playing experience.

[0134] In the prior art, when the error frame repair is performed at the playing end, the repair effect is poor due to the lack of accurate prediction and processing of the error frame, especially when the continuous packet loss leads to the continuous encoding frame loss, it is difficult to ensure the repair accuracy and the playing fluency at the same time.

[0135] Figure 4 The flowchart of the method for improving the video data playing quality provided by the embodiment of the application Figure Three , which is applied to the video playing end of the video streaming system, as shown in Figure 4 , the method comprises the following steps.

[0136] S401, receiving the second video data sent by the cloud, and performing the frame rate detection and the missing frame detection on the second video data;

[0137] The second video data is obtained by the cloud through the method for improving the video data playing quality applied to the cloud of the video streaming system, and the first video data received is repaired.

[0138] In the case of multiple missing frames of a video, the frame rate is not up to standard and the video playback appears to be stuck. Before playing the video, the video playback end first performs frame rate detection and missing frame detection on the second video data.

[0139] S402、In the case of multiple missing frames of a video, the frame rate is not up to standard and the video playback appears to be stuck. Before playing the video, the video playback end first performs frame rate detection and missing frame detection on the second video data.

[0140] In the case of multiple missing frames of a video, the frame rate is not up to standard and the video playback appears to be stuck. Before playing the video, the video playback end first performs frame rate detection and missing frame detection on the second video data.

[0141] In a possible implementation, the continuous frame compensation of the second video data is described in detail, including:

[0142] Based on the second video data, determine the image group, extract the spatio-temporal features in the video frames in the image group through the convolution layer of the diffusion model; capture the cross-frame dependency relationship of the spatio-temporal features through the time sequence attention layer, and establish the motion model of the objects in the image group; generate continuous frames through the diffusion model and the spatio-temporal features, the cross-frame dependency relationship and the motion model.

[0143] The diffusion model is trained based on sample data.

[0144] For the second video data with a number of missing video frames reaching the fifth preset threshold, determine the image group that needs to be compensated for continuous frames, extract the spatio-temporal features in the video frames in the image group through the convolution layer of the diffusion model trained in advance based on sample data; in the time sequence attention layer, expand the time dimension to the sequence length, capture the cross-frame dependency relationship through the multi-head attention mechanism, and establish the motion model of the objects; generate continuous frames through the diffusion model and the spatio-temporal features, the cross-frame dependency relationship and the motion model.

[0145] In a possible implementation, the video playback end includes a native back end and a front-end player; receives the second video data sent by the cloud end, and before performing frame rate detection and missing frame detection on the second video data, the method further includes:

[0146] The front-end player determines the connection object according to the mode information configured by the user, and the mode information includes a first playback mode and a second playback mode.

[0147] When the mode information is the first playing mode, the connection object is determined as the native backend, and the third video data sent by the native backend is received and played.

[0148] When the mode information is the second playing mode, the connection object is determined as the cloud, and the fourth video data sent by the cloud is received and played, wherein the fourth video data is obtained by encapsulating the first video data after the analysis processing.

[0149] It can be understood that, in the first playing mode, the native backend receives the second video data sent by the cloud, and performs continuous frame compensation in the case of multi-frame missing of the video to determine the third video data and send the third video data to the front-end player; in the second playing mode, the front-end player directly receives the fourth video data sent by the cloud.

[0150] The method for improving the playing quality of video data provided by the embodiment of the application receives the second video data sent by the cloud, performs frame rate detection and missing frame detection on the second video data, realizes multi-frame detection of the video, performs continuous frame compensation on the second video data when the number of missing frames of the video of the second video data reaches the fifth preset threshold, obtains the third video data, and plays the third video data, so that the video after the continuous frame compensation is visually coherent and natural, and the video is prevented from appearing to be stuck and jumping, the third video data is obtained by the cloud and the video playing end cooperatively processing, the video playing frame rate is guaranteed, the video playing is prevented from appearing to be full of screen and stuck, and a good playing experience is provided.

[0151] Figure 5 The structure diagram of the video streaming media system provided by the embodiment of the application Figure One As shown in the figure, Figure 5 The video streaming media system 50 provided by the embodiment of the application includes a video sending end 501 and a cloud 501, and the video sending end 501 and the cloud 501 are in communication connection.

[0152] The video sending end 501 is configured to send the first video data to the cloud 502.

[0153] The cloud is configured to receive the first video data sent by the video sending end, and perform error frame detection based on the first video data, and when an error frame is detected, perform repair processing on the received first video data by using the method for improving the playing quality of video data applied to the cloud of the video streaming media system.

[0154] The video streaming media system provided by the embodiment of the application can execute the method for improving the playing quality of video data applied to the cloud of the video streaming media system provided by the method embodiment, and the implementation principle and technical effects are similar, which will not be described here.

[0155] In the prior art, video data is processed in a single link, and full-link optimization cannot be formed. For example, small error frames that are not processed in the cloud are accumulated, which causes the video playback end to play with lag, and the video playback end lacks the support of the encoding layer of the original video in independent frame insertion.

[0156] Figure 6 A structural diagram of a video streaming system provided by an embodiment of the present application Figure Two In a possible implementation, the video streaming system further includes a video playback end 503, which is in communication connection with the cloud 502.

[0157] The video playback end performs continuous frame compensation on the received second video data by using the method for improving the video data playback quality of the video playback end of the video streaming system.

[0158] The video streaming system provided by the embodiment can perform the method for improving the video data playback quality of the video playback end of the video streaming system provided by the method embodiment, and the third video data obtained through the cooperation of the cloud and the video playback end guarantees the video playback frame rate, avoids the phenomenon of screen flashing and lagging in video playback, and provides a good playback experience.

[0159] Figure 7 A structural diagram of a video streaming system provided by an embodiment of the present application Figure Three In a possible implementation, as shown in Figure 7 The video playback end 503 includes a native back end 5031 and a front-end player 5032.

[0160] The native back end 5031 performs continuous frame compensation on the second video data, obtains third video data based on the second video data after the continuous frame compensation, and sends the third video data to the front-end player.

[0161] The front-end player 5032 receives and plays the third video data.

[0162] The video streaming system provided by the embodiment can perform the method for improving the video data playback quality of the video playback end of the video streaming system provided by the method embodiment, and the implementation principle and technical effects are similar, and details are not repeated here.

[0163] Figure 8 A structural diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 8 The electronic device 80 provided by the embodiment includes at least one processor 801 and a memory 802. Optionally, the device 80 further includes a communication component 803. The processor 801, the memory 802, and the communication component 803 are connected through a bus 804.

[0164] In the implementation process, the at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that the at least one processor 801 executes the method described above.

[0165] The implementation process of the processor 801 can refer to the implementation process of the method embodiments described above, which has similar implementation principles and technical effects, and details are not described here.

[0166] In the above embodiments, it should be understood that the processor can be a central processing unit (English: Central Processing Unit, CPU for short), and can also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, DSP for short), application specific integrated circuits (English: Application Specific Integrated Circuit, ASIC for short), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by hardware and software modules in the processor.

[0167] The memory can contain a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0168] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit only one bus or one type of bus.

[0169] The present application also provides a computer program product, comprising a computer program, which is executed by a processor to implement the method described above.

[0170] The present application also provides a computer-readable storage medium, which stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method described above is implemented.

[0171] The above-mentioned readable storage medium can be realized by any type of volatile or nonvolatile storage devices or their combinations, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0172] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0173] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0174] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0175] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0176] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0177] It can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various media that can store program codes.

[0178] Finally, it should be noted that: those skilled in the art will easily think of other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application that follow the general principles of the present application and include common knowledge or conventional technical means in the art that are not disclosed in the present application, and is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims.

Claims

1. A method for improving the quality of video data playback, characterized in that: The method is applied to a cloud of a video streaming system, wherein the video streaming system further includes a video transmitter, the cloud receives first video data sent by the video transmitter, and performs error frame detection based on the first video data. The method includes: When an error frame is detected, repairing the error frame based on slice data and macroblock data of the error frame to obtain a repaired frame; Based on the repaired frame, second video data is determined, and the second video data is sent to the video playback end after receiving a video acquisition request from the video playback end.

2. The method according to claim 1, characterized in that The repairing process of the error frame based on the slice data and macroblock data of the error frame to obtain a repaired frame includes: Obtaining network abstraction layer data of the error frame, and determining a frame type corresponding to the error frame based on the network abstraction layer data; Parsing the network abstraction layer data to obtain slice data of the error frame, and determining whether macroblock error correction can be performed on the error frame according to preset conditions, wherein the preset conditions include a slice data abnormality exceeding a first preset threshold and the slice data not conforming to a first syntax specification; If not, performing single-frame compensation on the error frame to obtain a repaired frame; If possible, based on the frame type and the macroblock data, macroblock check is performed on the macroblocks in the error frame to determine the abnormal macroblocks corresponding to the error frame, and macroblock error correction is performed on the abnormal macroblocks. Based on the corrected abnormal macroblocks, a repair frame is determined.

3. The method according to claim 2, characterized in that The frame type includes a first type, the macroblock data includes macroblock residual data and a motion vector, and the abnormal macroblock includes: any one of: a first abnormal macroblock whose macroblock data does not conform to a second syntax specification, a second abnormal macroblock whose macroblock residual data exceeds a second preset threshold, and a third abnormal macroblock whose motion vector exceeds a third preset threshold.

4. The method according to claim 3, characterized in that The performing macroblock error correction on the abnormal macroblock includes: When the frame type is the first type, determining that the abnormal macroblock is a normal macroblock in a preset area, and determining valid data corresponding to the motion vector of the normal macroblock and the amount of valid data; Determining whether the amount of valid data is less than a fourth preset threshold; If so, using the motion vector of the first associated macroblock corresponding to the abnormal macroblock in the reference frame as the motion vector of the abnormal macroblock, wherein the reference frame is determined based on the preceding reference frame index in the macroblock data of the abnormal macroblock; If not, repairing the motion vector of the abnormal macroblock based on the valid motion vector of the normal macroblock, and determining a second associated macroblock corresponding to the repaired abnormal macroblock in the reference frame; Based on the second associated macroblock, a search range is determined, a matching degree between the candidate macroblocks in the search range and the abnormal macroblock is calculated, and a matching macroblock is determined based on the matching degree, and the abnormal macroblock is replaced with the matching macroblock.

5. The method according to claim 3, characterized in that The frame type further includes a second type, and the performing macroblock error correction on the abnormal macroblock includes: When the frame type is the second type, extracting adjacent macroblocks of the abnormal macroblock; Performing edge detection on the adjacent macroblocks to determine weights corresponding to the adjacent macroblocks; Determining a second pixel value based on the weight and the first pixel value corresponding to the adjacent macroblock, and performing interpolation processing on the abnormal macroblock based on the second pixel value; For abnormal macroblocks after interpolation processing, texture completion optimization is performed to repair the area with missing texture.

6. The method according to claim 2, characterized in that The performing single-frame compensation on the error frame to obtain a repaired frame includes: Determine a preceding frame and a succeeding frame of the error frame, input the preceding frame and the succeeding frame into an optical flow model to obtain a forward optical flow corresponding to the preceding frame and a backward optical flow corresponding to the succeeding frame, wherein the optical flow model is trained based on sample frames; Performing motion compensation on each pixel of the error frame according to the forward optical flow and the backward optical flow to determine a corresponding mapped pixel value; A repaired frame is determined based on the mapped pixel value of each pixel.

7. The method according to claim 1, characterized in that The video streaming system further includes a video playback end, which includes a front-end player and a native back-end. Before repairing the error frame based on the slice data and macroblock data of the error frame to obtain the repaired frame, the method further includes: Receiving mode information sent by the video player, where the mode information is configured by the user on the front-end player and includes a first play mode and a second play mode; When the mode information is the first playback mode, repairing the received first video data, and sending the second video data to the native backend after receiving a video acquisition request sent by the native backend; When the mode information is the second playback mode, after receiving the video acquisition request sent by the video player, the fourth video data is sent to the front-end player, and the fourth video data is obtained by encapsulating the first video data after parsing.

8. A method for improving the quality of video data playback, characterized in that: Applied to a video playback terminal of a video streaming system, the method includes: receiving second video data sent by the cloud, and performing frame rate detection and frame missing detection on the second video data, wherein the second video data is obtained by the cloud by repairing the received first video data using the method according to any one of claims 1 to 6; When it is detected that the number of missing video frames of the second video data reaches a fifth preset threshold, continuous frame compensation is performed on the second video data, and third video data is obtained based on the second video data after continuous frame compensation, and the third video data is played.

9. The method according to claim 8, characterized in that The performing continuous frame compensation on the second video data includes: Determining an image group based on the second video data, and extracting spatiotemporal features within video frames within the image group using a convolutional layer of a diffusion model, wherein the diffusion model is trained based on the sample data; Capturing cross-frame dependencies of the spatiotemporal features through a temporal attention layer and building a motion model of objects in the image group; Continuous frames are generated through the diffusion model, the spatiotemporal features, the cross-frame dependencies, and the motion model.

10. The method according to claim 8, characterized in that The video playback end includes a native backend and a front-end player; before receiving the second video data sent by the cloud and performing frame rate detection and frame missing detection on the second video data, the method further includes: The front-end player determines a connection object according to mode information configured by the user, wherein the mode information includes a first play mode and a second play mode; Wherein, when the mode information is the first playback mode, determining that the connection object is a native backend, and receiving and playing the third video data sent by the native backend; When the mode information is the second playback mode, the connection object is determined to be the cloud, and fourth video data sent by the cloud is received and played. The fourth video data is obtained by encapsulating the first video data after analysis.

11. A video streaming system, characterized in that: include: A video sending end and a cloud, wherein the video sending end is communicatively connected with the cloud; The video sending end is used to send the first video data to the cloud; The cloud receives the first video data sent by the video sending end, and performs error frame detection based on the first video data. When an error frame is detected, the received first video data is repaired by the method according to any one of claims 1 to 6.

12. The video streaming system according to claim 11, characterized in that: The video streaming system further includes: a video playback terminal, the video playback terminal being communicatively connected to the cloud; The video playback end performs continuous frame compensation on the received second video data by using the method according to any one of claims 8 to 11.

13. The video streaming system according to claim 12, wherein: The video player includes a native backend and a front-end player; The native backend performs continuous frame compensation on the second video data, obtains third video data based on the second video data after continuous frame compensation, and sends the third video data to the front-end player; The front-end player receives and plays the third video data.

14. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method according to any one of claims 1 to 10.

Images