Multimedia data processing method, electronic device and storage medium

Abstract

The present application discloses a multimedia data processing method, which is applied to the acquisition end, including: generating a target audio frame and a target video frame according to the multimedia data to be processed; determining a first timestamp of the target audio frame and a second timestamp of the target video frame; transmitting the first timestamp, the second timestamp, the target audio frame, and the target video frame to the client together; the first timestamp and the second timestamp are used for the client to determine whether a predetermined abnormality occurs, wherein the predetermined abnormality includes at least one of the following: transmission desynchronization between the target audio frame and the target video frame, transmission abnormality of at least one of the target audio frame and the target video frame, and jamming of the multimedia data playback. In this way, in the field of Internet technology that applies real-time transmission of multimedia data, the target audio frame and the target video frame are transmitted to the client together with the corresponding timestamp, and the client can accurately evaluate the quality of multimedia data transmission and playback according to the moment indicated by the timestamp.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a multimedia data processing method, electronic equipment and storage medium. Background Art

[0002] Currently, in scenarios involving real-time video transmission, such as live broadcasting, the acquisition end needs to collect live broadcast data, encode and encapsulate the data, transmit it through the network, and finally use a player on the user end to decode and play the data.

[0003] The quality of live video is affected by many factors such as delay, freeze, and audio and video asynchrony. In the prior art, the test of delay, freeze, and audio and video asynchrony often requires the use of specific test equipment to test the audio and video data, or the use of subjective judgment and manual operation. For example, to test the delay of live broadcast, it is necessary to start a timer at the acquisition end and the playback terminal at the same time, and include the source video and the playback video in one photo through screen capture or mobile phone photography, and finally calculate the time difference. Testing live broadcast freezes requires manual supervision to subjectively judge the number of freezes per unit time to calculate the freeze rate; testing audio and video synchronization requires decomposing the video and audio streams, and checking the time difference between the beginning of the image and audio for evaluation.

[0004] However, this method is limited by manpower and has many external interference factors, which can easily lead to inaccurate calculations and large errors in the quality assessment of audio and video data. Summary of the invention

[0005] In view of this, the present invention provides a multimedia data processing method, an electronic device and a storage medium.

[0006] The technical solution of the present invention is achieved in this way:

[0007] In a first aspect, an embodiment of the present invention provides a multimedia data processing method, which is applied to a collection end, and the method includes:

[0008] Generate a target audio frame and a target video frame according to the multimedia data to be processed;

[0009] Determining a first timestamp of the target audio frame and a second timestamp of the target video frame;

[0010] The first timestamp, the second timestamp, the target audio frame, and the target video frame are transmitted to the client together; the first timestamp and the second timestamp are used by the client to determine whether a predetermined abnormality occurs, wherein the predetermined abnormality includes at least one of the following: transmission desynchronization between the target audio frame and the target video frame, transmission abnormality of at least one of the target audio frame and the target video frame, and playback of the multimedia data is blocked.

[0011] Furthermore, the first timestamp and the second timestamp are carried in an information frame and transmitted to the client.

[0012] Furthermore, the information frame includes: a supplemental enhancement information SEI frame; and each group of pictures GOP where the target video frame is located includes one SEI frame.

[0013] Furthermore, the method further comprises:

[0014] Determining a data type of the multimedia data;

[0015] If the data type is audio data, determining that the moment when the audio data is acquired is the first moment indicated by the first timestamp, and writing information of the first moment into a first register; the information of the first moment written into the first register is used for generating the first timestamp;

[0016] If the data type is video data, determine that the moment when the video data is acquired is the second moment indicated by the second timestamp, and write the information of the second moment into the second register; write the information of the first moment into the second register for generating the second timestamp.

[0017] Furthermore, the method further comprises:

[0018] When determining the first timestamp, reading the information of the first moment from the first register, and resetting a flag bit of the first register, wherein the flag bit of the first register indicates that the first register is set to a preset value;

[0019] When determining the second timestamp, the second time information is read from the second register.

[0020] In a second aspect, an embodiment of the present invention provides a multimedia data processing method, which is applied to a client, and the method includes:

[0021] Obtain a target audio frame, a target video frame, a first timestamp, and a second timestamp;

[0022] Determine whether a predetermined abnormality occurs based on the first timestamp and the second timestamp; the predetermined abnormality includes at least one of the following: transmission desynchronization between the target audio frame and the target video frame, transmission abnormality of at least one of the target audio frame and the target video frame, and freeze in the playback of multimedia data.

[0023] Furthermore, the method further comprises:

[0024] If the predetermined exception occurs, an exception message is sent to the server.

[0025] Furthermore, the method further comprises:

[0026] determining a third timestamp indicating a time instant of receiving the target audio frame, and determining a fourth timestamp indicating a time instant of receiving the target video frame;

[0027] The determining, according to the first timestamp and the second timestamp, whether a predetermined abnormality occurs includes:

[0028] Determine a first time difference between the first timestamp and the moment indicated by the second timestamp, and determine a second time difference between the moment indicated by the third timestamp and the moment indicated by the fourth timestamp;

[0029] determining a difference between the first time difference and the second time difference;

[0030] If the difference is less than or equal to a first threshold, determining that the target audio frame is synchronized with the target video frame;

[0031] If the difference is greater than the first threshold, it is determined that the target audio frame and the target video frame are out of sync.

[0032] Further, the determining whether a predetermined abnormality occurs according to the first timestamp and the second timestamp further includes:

[0033] Determine the time difference between the second timestamp and the moment indicated by the fourth timestamp as the transmission delay, and / or determine the time difference between the first timestamp and the moment indicated by the third timestamp as the transmission delay;

[0034] Determine an average delay based on transmission delays of multiple target video frames;

[0035] Determining whether the predetermined abnormality occurs according to the average delay;

[0036] The determining, based on the average delay, whether the predetermined abnormality occurs includes at least one of the following:

[0037] If the average delay is greater than a second threshold, determining that the transmission delay is abnormal;

[0038] If the difference between the transmission delay and the average delay is greater than a third threshold, it is determined that a freeze occurs in the multimedia data playback.

[0039] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising: a processor and a memory for storing a computer program that can be run on the processor;

[0040] When the processor runs the computer program, it executes the steps of the method described in one or more of the above technical solutions.

[0041] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer-executable instructions; after the computer-executable instructions are executed by a processor, the method described in one or more of the aforementioned technical solutions can be implemented.

[0042] The multimedia data processing method provided by an embodiment of the present invention is applied to an acquisition end, including: determining a first timestamp of a target audio frame and a second timestamp of a target video frame; transmitting the first timestamp, the second timestamp, the target audio frame, and the target video frame to a client together; the first timestamp and the second timestamp are used by the client to determine whether a predetermined abnormality occurs, wherein the predetermined abnormality includes at least one of the following: transmission desynchronization between the target audio frame and the target video frame, transmission abnormality of at least one of the target audio frame and the target video frame, and freeze in the playback of the multimedia data.

[0043] In this way, the acquisition end can record and provide the exact time to the client based on the timestamp. The client can determine the time indicated by the timestamp corresponding to the target audio frame and the target video frame based on the first timestamp and the second timestamp sent by the acquisition end, and then more accurately determine the transmission anomaly, playback anomaly and other situations based on the moment, without the need for manual judgment of the anomaly, and can greatly improve the accuracy of anomaly locating. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a multimedia data processing method provided by an embodiment of the present invention;

[0045] Figure 2 A flowchart of a multimedia data processing method provided by an embodiment of the present invention;

[0046] Figure 3 A flowchart of a multimedia data processing method provided by an embodiment of the present invention;

[0047] Figure 4A flowchart of a multimedia data processing method provided by an embodiment of the present invention;

[0048] Figure 5 A flowchart of a multimedia data processing method provided by an embodiment of the present invention;

[0049] Figure 6 A flowchart of a multimedia data processing method provided by an embodiment of the present invention;

[0050] Figure 7 A schematic diagram of the structure of a video frame sequence provided by an embodiment of the present invention;

[0051] Figure 8 A schematic diagram of a dual timestamp encoding process provided by an embodiment of the present invention;

[0052] Fig. 9 A schematic diagram of a structure for storing a timestamp in a frame sequence provided by an embodiment of the present invention;

[0053] Fig.10 A schematic diagram of a relationship between predetermined abnormal timestamps provided in an embodiment of the present invention;

[0054] Fig.11 A schematic diagram of the relationship between timestamps when audio and video are out of sync provided by an embodiment of the present invention;

[0055] Fig.12 A schematic diagram of the structure of a device for dynamically evaluating the quality of online live broadcast provided by an embodiment of the present invention;

[0056] Fig.13 A schematic diagram of the change in live broadcast delay provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0058] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0059] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0061] like Figure 1 As shown, an embodiment of the present invention provides a multimedia data processing method, which is applied to a collection end, and the method includes:

[0062] S110: generating a target audio frame and a target video frame according to the multimedia data to be processed;

[0063] S120: Determine a first timestamp of the target audio frame and a second timestamp of the target video frame;

[0064] S130: Transmit the first timestamp, the second timestamp, the target audio frame, and the target video frame together to the client; the first timestamp and the second timestamp are used by the client to determine whether a predetermined abnormality occurs.

[0065] The predetermined abnormality may include at least one of the following: transmission desynchronization between the target audio frame and the target video frame, transmission abnormality of at least one of the target audio frame and the target video frame, and playback of the multimedia data being interrupted.

[0066] Here, the multimedia data may be multimedia information content that needs to be transmitted to the client for playback, such as live video recorded at the live end, etc. The multimedia data may include audio data representing sound information and / or video data representing picture information.

[0067] The acquisition end can be used to collect multimedia data and related parameters. For example, the acquisition end can be a user-end device that initiates live broadcasting, or an application, server, cloud platform, etc. for collecting and processing multimedia data. The client can be used to receive and play multimedia data sent by the acquisition end. For example, the client can be a viewer-end device for watching live broadcasting, or an application, server, cloud platform, etc. on the viewer side. After receiving the target audio frame and the target video frame, the client can output multiple target audio frames and target video frames in the order of timestamps to play the complete multimedia data.

[0068] In an embodiment of the present invention, target audio frames and target video frames are generated according to the multimedia data to be processed, and the multimedia data can be encoded, including: decomposing the audio data in the multimedia data into at least one target audio frame, for example, dividing the audio data into multiple target audio frames according to a certain sampling frequency, frame rate, etc.; decomposing the video data into at least one target video frame. The encoding of the video data can adopt encoding technologies such as H.264 and H.265.

[0069] In one embodiment, determining the first timestamp of the target audio frame and the second timestamp of the target video frame may include: determining the first timestamp corresponding to the target audio frame, for example, generating the first timestamp according to the moment when the audio data is acquired, or generating the first timestamp according to the moment when the audio data is encoded to generate the target audio frame; determining the second timestamp of the target video frame, for example, generating the second timestamp according to the moment when the video data is acquired, or generating the second timestamp according to the moment when the video data is encoded to generate the target video frame.

[0070] In another embodiment, the predetermined exception may be an abnormal situation that affects the playback effect when the client plays and outputs video data and audio data based on the received target video frames and target audio frames, for example, it may include: transmission desynchronization between the target audio frame and the target video frame, that is, the phenomenon of audio and video being out of sync during playback; transmission abnormality of at least one of the target audio frame and the target video frame, such as excessive transmission delay of the target video frame; and / or freezes in the playback of multimedia data.

[0071] In this way, the required moment is accurately recorded through the timestamp, and the timestamp is transmitted to the client together with the target video frame and the target audio frame. The client can accurately determine the moment indicated by the timestamp based on the timestamp, and then more accurately determine the transmission anomaly, playback anomaly, etc. based on the moment. Based on this, the client does not need to manually judge the anomaly, which effectively improves the intelligence of multimedia data transmission and playback quality assessment, and can greatly improve the accuracy of client anomaly positioning based on the absolute time recorded by the timestamp.

[0072] In one embodiment, the first timestamp may be carried in the frame header of the target video frame; the second timestamp may be carried in the frame header of the target audio frame. In this way, each video frame and audio frame of the target video may be inserted with a corresponding timestamp, which may be used to implement frame-by-frame out-of-sync abnormality determination.

[0073] In some embodiments, the first timestamp and the second timestamp are carried in an information frame transmitted to the client.

[0074] In the embodiment of the present invention, the information frame may be a companion frame generated when encoding video data and generating a target video frame for recording target video frame information, or may be other forms of frame data. The information frame may correspond to the target video frame one-to-one, and the acquisition end may transmit the information frame carrying the first timestamp and the second timestamp together with the target audio frame and the target video frame to the client.

[0075] In this way, by encoding the multimedia data to form an information frame that carries the first timestamp and the second timestamp, it is possible to provide the client with a timestamp indicating the absolute time, while suppressing the changes in the frame data structure caused by carrying the timestamp in the target video frame and the target audio frame, thereby avoiding the client data processing efficiency from becoming lower, and suppressing the first timestamp and the second timestamp from being directly packaged with the target video frame and the target audio frame, resulting in the data being more scattered and easy to lose.

[0076] In some embodiments, the information frame may include: a supplemental enhancement information SEI frame; and each group of pictures GOP where the target video frame is located includes one SEI frame.

[0077] In an embodiment of the present invention, the video data is encoded, and the generated video frame sequence may include multiple groups of pictures (Group of Pictures, GOP), each GOP includes a key frame I (Intra-frame) frame, at least one B frame, at least one P frame and a Supplemental Enhancement Information (SEI) frame. Here, the B frame (bi-directional interpolated prediction frame) is a bidirectional reference frame used to record the changes of the I frame in the GOP compared with the I frames in the previous and next GOPs; the P frame (Predictive-frame) is a forward reference frame used to record the changes of the I frame in the GOP compared with the I frame in the previous GOP.

[0078] Here, the I frame is the target video frame. After generating the target video frame, the acquisition end uses the target video frame as the I frame to generate the corresponding GOP, and then generates a first timestamp and a second timestamp, and records them in the SEI frame of the GOP.

[0079] In one embodiment, the information frame may be a SEI frame, which may record other parameters of the target video frame, such as parameters of an encoder for encoding video data, picture parameters of the target video frame, etc., for the client to perform higher quality multimedia data synthesis and output playback based on the parameters of the target video frame.

[0080] In this way, the SEI frame formed based on video data encoding directly carries the first timestamp and the second timestamp, and can make full use of the recording function of the SEI frame. There is no need to additionally package the first timestamp and the second timestamp, and there is no need to write the timestamp into the frame data, which will cause damage to the target video frame and the target audio frame data structure.

[0081] In some embodiments, Figure 2 As shown, the method also includes:

[0082] S101: Determine the data type of the multimedia data;

[0083] S102: If the data type is audio data, determine that the moment when the audio data is acquired is the first moment indicated by the first timestamp, and write information of the first moment into a first register; the information of the first moment written into the first register is used for generating the first timestamp;

[0084] S103: If the data type is video data, determine that the moment when the video data is acquired is the second moment indicated by the second timestamp, and write the information of the second moment into a second register; the information of the first moment written into the second register is used for generating the second timestamp.

[0085] In an embodiment of the present invention, after the audio data and the video data are acquired, the information of the first moment and the information of the second moment are respectively saved by the first register and the second register to ensure that the timestamp is not affected by the time delay when it is generated, and the moment when the audio data and the video data are acquired can still be accurately acquired.

[0086] In one embodiment, the time when the audio data is acquired and the time when the video data is acquired may be determined by establishing synchronization with a local clock, or may be determined by acquiring corresponding time via a wireless local area network (Wireless Local Area Network, WLAN).

[0087] In another embodiment, the second register may be a register that is synchronized with a clock through WLAN or other means and is used to record and update the time in real time. After determining that the data type of the multimedia data is video data, the current time in the second register is directly read to generate a second timestamp. If the data type of the multimedia data is determined to be audio data, the current time information in the second register may be read and stored in the first register as the first time information, and the first time information is read from the first register when generating the first timestamp.

[0088] It is understandable that the register may also be replaced by a removable hard disk, magnetic disk or other device with read, write and storage functions.

[0089] In this way, by storing the time information when the target audio frame and the target video frame are acquired respectively through two registers, the information of the first time and the second time can be saved in time to suppress the error of time delay when generating the timestamp, and it can fully adapt to the performance of terminal hardware encoding and suppress the storage of data in only one register, which will cause the data storage to be too crowded and affect the multimedia data transmission performance.

[0090] In some embodiments, the method further comprises:

[0091] When determining the first timestamp, reading the information of the first moment from the first register, and resetting a flag bit of the first register, wherein the flag bit of the first register indicates that the first register is set to a preset value;

[0092] When determining the second timestamp, the second time information is read from the second register.

[0093] In an embodiment of the present invention, after determining that the data type of multimedia data is audio data, the first moment information therein is read from the first register, a first timestamp is generated based on the first moment, and the first register is reset to a preset value, for example, reset to "0". Since the sampling rate, frame rate, etc. of the target audio frame generated by encoding the audio data are not completely consistent with the sampling rate, frame rate, etc. of the video data encoding, the generation rate of the target audio frame and the target video frame are not consistent, and there may be a situation where multiple target video frames are continuously generated after sampling to generate one target audio frame. Therefore, if the first register is not reset in time after reading the first moment information from the first register, when the first timestamp and the second timestamp are determined for the subsequent multiple target video frames, the first moment information will continue to be read from the first register to generate repeated first timestamps, resulting in an error between the timestamp and the actual situation.

[0094] In this way, it can be indicated that the first moment information corresponding to the current target audio frame has been retrieved and the corresponding first timestamp has been generated, which prevents the first moment information from being stored in the first register for a long time, resulting in subsequent continuous retrieval and generation of repeated first timestamps, thereby improving the accuracy of timestamp generation. On this basis, the continuous accumulation and storage of different first moment information by the first register can also be reduced, thereby improving the performance of the first register.

[0095] like Figure 3 As shown, an embodiment of the present invention provides a multimedia data processing method, which is applied to a client, and the method includes:

[0096] S210: Acquire a target audio frame, a target video frame, a first timestamp, and a second timestamp;

[0097] S220: Determine whether a predetermined abnormality occurs based on the first timestamp and the second timestamp; the predetermined abnormality includes at least one of the following: transmission desynchronization between the target audio frame and the target video frame, transmission abnormality of at least one of the target audio frame and the target video frame, and freeze in the playback of multimedia data.

[0098] In an embodiment of the present invention, the target audio frame, target video frame, first timestamp and second timestamp sent by the acquisition end can be received, or the target audio frame, target video frame and information frame sent by the acquisition end can be received, and the first timestamp and second timestamp carried in the information frame can be extracted.

[0099] Exemplarily, when the acquisition end transmits the target audio frame and video frame sequence to the client, the SEI frame in each GOP is determined from the video frame sequence, and the first timestamp and the second timestamp are extracted from the SEI frame, and the I frame in the GOP is determined to be the target video frame.

[0100] In one embodiment, the acquisition time of the audio data and video data corresponding to the target audio frame and the target video frame can be determined respectively according to the first timestamp and the second timestamp, and then the decoding time of the audio data and the video data obtained by the client for decoding the target audio frame and the target video frame can be determined, so as to determine whether there are predetermined abnormalities in the audio data and the video data during playback based on the decoding time and the acquisition time in combination with different judgment methods.

[0101] In this way, the client can more accurately and quickly determine whether there is an abnormality based on the absolute time indicated by the first timestamp and the second timestamp generated by the acquisition end. At the same time, the client does not need to manually determine the relevant values ​​of abnormal conditions such as audio and video out of sync and freeze, thereby improving the accuracy and intelligence of the client's abnormality detection.

[0102] In some embodiments, Figure 4 As shown, the method also includes:

[0103] S230: If the predetermined exception occurs, an exception message is sent to the server.

[0104] In an embodiment of the present invention, after the client determines that a predetermined exception exists based on the moments indicated by the first timestamp and the second timestamp, it can generate exception information indicating the existence of the exception, such as an exception handling request, exception report information, etc., and send the exception information to a server, or an application control platform, etc.

[0105] In this way, the server can process the abnormal conditions in the current audio data and video data in a timely manner based on the abnormal information, quickly eliminate the abnormal conditions that affect the playback effect, such as audio and video asynchrony and freeze, and timely optimize the playback quality of multimedia data.

[0106] In some embodiments, Figure 5 As shown, the method also includes:

[0107] S201: Determine a third timestamp indicating a receiving moment of the target audio frame, and determine a fourth timestamp indicating a receiving moment of the target video frame;

[0108] The S220 may include:

[0109] S221: Determine a first time difference between the moments indicated by the first timestamp and the second timestamp, and determine a second time difference between the moments indicated by the third timestamp and the fourth timestamp;

[0110] S222: Determine a difference between the first time difference and the second time difference;

[0111] S223: If the difference is less than or equal to a first threshold, determining that the target audio frame is synchronized with the target video frame;

[0112] S224: If the difference is greater than the first threshold, determine that the target audio frame and the target video frame are out of sync.

[0113] Here, the target audio frame and the target video frame are out of sync, that is, the target audio frame and the target video frame are out of sync.

[0114] In the embodiment of the present invention, after the client obtains the target audio frame and the target video frame, it starts decoding to obtain the audio data and the video data, so the third timestamp can be used to indicate the receiving time of the target audio frame. If the decoding is performed immediately after the receiving, the third timestamp can also be used to indicate the decoding time when the decoding of the target audio frame starts. The same is true for the fourth timestamp.

[0115] In one embodiment, the first time stamp aet of the target audio frame is determined. i Here, a represents audio, e represents encoding, and t i Represents the time corresponding to the i-th frame. It can be determined that aet i Is it 0? If aet i 0 indicates that the first register has been reset to 0 when the first timestamp is generated at the acquisition end, so the extracted first time information is 0, and the generated first timestamp is also 0, that is, the current target video frame has no corresponding target audio frame when it is generated. At this time, the determination process from S221 to S224 is not performed.

[0116] In another embodiment, if aet i If it is not 0, then according to the formula eint i =vet i -aet i Calculate AET i The second timestamp vet of the target video frame i The first time difference between the indicated moments eint i , which is used to represent the time interval between the target video frame and the target audio frame obtained at the acquisition end, or the encoding time interval between the target video frame and the target audio frame. Here, v represents the video video.

[0117] In yet another embodiment, a third timestamp aet indicating the time when the target audio frame is received is determined. i , where d represents decoding decode, and the fourth timestamp vdt indicating the moment when the target video frame is received i Therefore, according to the formula aint i =vdt i -adt i Determine the second time difference aint i , used to represent the reception time interval of the target video frame and the target audio frame at the client, or the decoding time interval of the target video frame and the target audio frame.

[0118] In one embodiment, after determining the third timestamp and the fourth timestamp, the first timestamp and the second timestamp can be saved in a third register of the client, and / or the third timestamp and the fourth timestamp can be saved in a fourth register of the client to perform timestamp recording and retain traces of the multimedia data decoding process.

[0119] Based on the formula difft i =aint i -eint i Calculate the difference between the first time difference and the second time difference difft i , and the difference difft i Compare with the preset first threshold. If difft i If it is greater than the first threshold, it may indicate that the decoding time interval between the target audio frame and the target video frame has changed too much compared to the encoding time interval. When the corresponding audio data and video data are output for playback, an abnormal situation of audio and video desynchronization will occur, such as the audio and picture being out of sync during live video playback.

[0120] In another embodiment, after the difference between the first time difference and the second time difference is calculated, the difference difft iUpload to the server for recording. i Determine if there is an audio or video out-of-sync abnormality. i It is carried in the exception information and sent to the server for the server to determine how to adjust the playback parameters of the audio data and / or video data, such as adjusting the frame rate, playback speed and other parameters to synchronize the playback of the audio and video data.

[0121] In this way, the encoding and decoding times of the acquisition end and the client are accurately indicated based on the timestamp, so that the synchronization of the audio and video frames can be calculated more intelligently and accurately, and then the abnormal situation of audio and video being out of synchronization can be quickly detected.

[0122] In some embodiments, Figure 6 As shown, the S220 further includes:

[0123] S225: Determine the time difference between the second timestamp and the moment indicated by the fourth timestamp as the transmission delay, and / or determine the time difference between the first timestamp and the moment indicated by the third timestamp as the transmission delay;

[0124] S226: Determine an average delay based on the transmission delays of the multiple target video frames;

[0125] S227: Determine whether the predetermined abnormality occurs according to the average delay;

[0126] The determining, based on the average delay, whether the predetermined abnormality occurs includes at least one of the following:

[0127] If the average delay is greater than a second threshold, determining that the transmission delay is abnormal;

[0128] If the difference between the transmission delay and the average delay is greater than a third threshold, it is determined that a freeze occurs in the multimedia data playback.

[0129] In an embodiment of the present invention, the time interval between the encoding time and the decoding time of the target video frame, that is, the transmission delay of the target video frame, can be determined according to the second timestamp and the fourth timestamp of the target video frame, or the transmission delay of the target audio frame can be determined based on the first timestamp and the third timestamp.

[0130] In one embodiment, according to the formula lat i =vdt i -vet i Determine the transmission delay lat of the target video frame i , and then based on the transmission delay of n target video frames, according to the formula Calculate the average delay avg_lat within a time period. The average delay can be used as a basis for evaluating the video data delay within a certain time period.

[0131] In one embodiment, the average delay avg_lat of the target video frame can be compared with a preset second threshold value. If avg_lat is greater than the second threshold value, it indicates that the delay of the video data in the time period is large, and the transmission delay of the target video frame is determined to be abnormal. If avg_lat is less than or equal to the second threshold value, it indicates that the delay of the video data in the time period is small, and the transmission delay of the target video frame is determined to be normal.

[0132] It is understandable that after determining that there is a transmission delay anomaly, the transmission delay of the corresponding multiple target video frames and the fourth timestamp, and / or the average delay can be uploaded to the server for recording, for example, carried in the abnormal information and reported to the server.

[0133] In another embodiment, according to the formula seq_t i =lat i -avg_lat, determine the delay difference seq_t between the transmission delay of each target video frame and the average delay i . Set the delay difference seq_t i Compare with the third threshold, if seq_t i If it is greater than the third threshold, it indicates that the transmission delay of the corresponding target video frame is higher than the average delay level of multiple target video frames, that is, during the video data output and playback process, an abnormal situation of freeze will occur at the location of the target video frame.

[0134] It can be understood that after determining whether there is a stuttering exception, the target video frame causing the stuttering, that is, the transmission delay and the fourth timestamp of the target video frame whose delay difference is greater than the third threshold, can be uploaded to the server for recording, for example, carried in the exception information and reported to the server.

[0135] In another embodiment, if the acquisition end transmits the target video frame to the client in the form of an I frame in a GOP, there may be an abnormal situation that the I frame in the GOP received by the client, that is, the target video frame is lost. Since the client only needs to extract the timestamp in the SEI frame of the GOP to determine whether the predetermined abnormality exists, if the I frame is lost, it cannot be discovered in time. Therefore, the transmission delay lat of multiple target video frames is determined based on the timestamp. i If it is determined that the transmission delay of a target video frame is lower than a preset frame loss standard value, or the reduction in the transmission delay compared to the previous target video frame exceeds a preset value, or is a negative number, it can be determined that the target video frame, i.e., the I frame in the GOP, is lost.

[0136] It is understandable that after determining that there is a frame loss exception, the timestamp and transmission delay corresponding to the GOP of the frame loss can be uploaded to the server, for example, can be carried in the exception information and reported to the server.

[0137] A specific example is provided below in combination with any of the above embodiments:

[0138] Since the acquisition end takes a short time to collect data, this embodiment adds a timestamp in the video encoding stage. In the encoding stage, the audio and video timestamps are placed in the SEI frame to mark each GOP. The encoded frame sequence reaches the client after passing through the streaming media platform. When the client plays, it is decoded to obtain the encoding timestamp, and then compared with the timestamp obtained in real time with the target video frame, so that end-to-end online live video quality assessment can be achieved based on absolute time. In the encoding stage, the method of adding timestamps to video and audio adopts hardware encoding. Hardware encoding can be but not limited to register reading and writing. The register reading and writing method is not only fast, but also has lower cost.

[0139] like Figure 7 As shown, during the encoding stage, the audio and video timestamps (Time Stamp) are placed in the SEI frame to mark each GOP.

[0140] like Figure 8 As shown, the embodiment of the present invention proposes a method and device for dynamically evaluating the quality of online live broadcast, which specifically relates to a dual timestamp video encoding technology, including the following steps:

[0141] Step 1: In the clock synchronization module, register A performs time synchronization operation from WLAN, and register A retains the timestamp.

[0142] Step 2: Determine the data source and write the timestamp to the register.

[0143] Specifically, it is determined whether the data source is video data or audio data, wherein the streaming media data includes video and audio data. If it is audio data, the timestamp in A is written into register B. If it is video data, the operation of step 3 is performed.

[0144] Step 3: Read the timestamp from the register and write it into the frame sequence.

[0145] When encoding a video frame, read the timestamp in register A, and when encoding an audio frame, read the timestamp in register B, forming a double timestamp, write the double timestamp into the video frame SEI, and proceed to step 4.

[0146] Step 4: Reset the audio data register to 0.

[0147] Specifically, each time the timestamp in register B is taken, register B needs to be reset to 0, which can be used as a basis for determining whether the audio encoding timestamp exists. The purpose is to prevent the next data from being video data, which may cause inaccurate calculation of the subsequent audio and video encoding timestamp.

[0148] Step 5: End encoding and complete subsequent packaging and other tasks.

[0149] like Fig. 9 As shown, an embodiment of the present invention provides a video timestamp encoding storage structure, wherein each timestamp occupies 8 bytes.

[0150] like Fig.10 As shown in the figure, from front to back, they are the encoding and decoding timestamp relationship diagrams under the conditions of delay, freeze and frame loss, where ta1 and ta2 are the encoding timestamp and decoding timestamp of video frame A, tb1 and tb2 are the encoding timestamp and decoding timestamp of audio frame B, and so on. Fig.11 The figure shows the relationship between encoding and decoding timestamps when the audio and video are not synchronized. The embodiment of the present invention provides a calculation method for dynamic evaluation of online live broadcast quality, and the following test quality index acquisition scenarios are provided:

[0151] 1) Calculation of delay, freeze, and I-frame loss

[0152] Specifically, when decoding at the playback end, extract the SEI frame and obtain the current decoding timestamp vdt i , extract the video encoding timestamp vet i , calculate the video encoding timestamp vet i With the decoded timestamp vdt i The time difference is the live broadcast delay of the video.

[0153] Specifically, calculate the instantaneous delay lat i As shown in formula (1-1):

[0154] lat i =vdt i -vet i Formula (1-1)

[0155] The i in formula (1-1) refers to the number of I frames from which the timing starts. i Refers to the delay value of the i-th video frame.

[0156] Specifically, the average delay avg_lat is calculated as shown in formula (1-2):

[0157]

[0158] Among them, i in formula (1-2) refers to the I-th frame from which timing starts, n represents the number of sampled key frames, and avg_lat refers to the average delay accumulated over i frames.

[0159] Furthermore, the difference between the instantaneous delay and the average delay of a frame is calculated as seq_t i , and obtain the difference sequence, which is shown in formula (1-3):

[0160] seq_t i =lat i -avg_lat formula (1-3)

[0161] Specifically, both freeze and I-frame loss are determined based on statistical analysis of delay data. If freeze occurs in the video, the instantaneous delay will increase, and if I-frame loss occurs in the video, the instantaneous delay will decrease or become negative. The determination of freeze and I-frame loss will be described with specific embodiments, wherein the determination of whether a frame is lost can only determine the loss of key frames, and cannot determine the loss of B frames and P frames.

[0162] 2) Calculation of audio and video asynchrony

[0163] Specifically, when decoding at the playback end, extract the video encoding timestamp vet in the SEI frame i , extract the audio encoding timestamp aet i , first of all i Is it 0? If it is 0, no calculation is performed. i If it is not 0, the time interval between the video frame and the audio frame is calculated. i Specifically, the calculation formula is as shown in formula (1-4):

[0164] eint i =vet i -aet i Formula (1-4)

[0165] Here, i in formula (1-4) refers to the number of I frames.

[0166] Furthermore, the video decoding timestamp generated when decoding the video frame obtains the actual audio and video time interval aint i for:

[0167] aint i =vdt i -adt i Formula (1-5)

[0168] Further, the audio-video synchronization difference difft is calculated i for:

[0169] difft i =aint i -eint i Formula (1-6)

[0170] Further, calculate the average audio and video synchronization difference avg_difft i for:

[0171]

[0172] Among them, i refers to the number of I frames from which the timing starts, avg_difft i It refers to the average delay of the accumulated i frames, and n represents the number of sampled key frames.

[0173] Specifically, the method provided in the embodiment of the present invention can be executed in a mobile terminal, a computer terminal or a similar computing device. For example, if the method is executed in a computer terminal, Fig.12 As shown, the computer terminal needs to rely on other equipment and devices to operate, and the other equipment includes: clock synchronization equipment, storage equipment, and the other devices include: acquisition equipment, encoding equipment, transmission equipment, decoding equipment, calculation equipment, and judgment equipment.

[0174] 1. The codec module includes an acquisition device, an encoding device, a transmission device and a decoding device.

[0175] The P01 acquisition device can be but is not limited to hardware acquisition devices such as cameras. The acquisition device acquires original video data and sends the generated data to the encoding device.

[0176] P02 encoding device, the present invention puts the clock synchronization device into the encoding device, and uses a storage device to store the timestamp generated by the clock synchronization device, wherein the storage device is divided into a first storage device and a second storage device, and the storage device can be but not limited to a register, a removable hard disk, a disk and other devices with read, write and storage functions. The device is used to complete the following steps:

[0177] S01: The clock synchronization device connects to the Internet to obtain a timestamp and stores the timestamp in a first storage device.

[0178] S02: Determine the data source and write the timestamp to the storage device.

[0179] Specifically, it is determined whether the data source is video data or audio data, wherein the video data contains audio data. If it is audio data, the timestamp in the first storage device is written to the second storage device and step S03 is performed; if it is video data, step S04 is performed.

[0180] S03: The second storage device is reset.

[0181] Specifically, a reset operation needs to be performed after reading the timestamp in the second storage device in order to prevent the next data from being video data and causing inaccuracy in the subsequent calculation of the audio and video time difference.

[0182] S04: Read the timestamp from the storage device and write it into the frame sequence.

[0183] When encoding a video frame, the timestamp in the first storage device is read, and when encoding an audio frame, the timestamp in the second storage device is read to form a double timestamp, which is written to the video and audio frame sequences respectively.

[0184] S05: inserting a double timestamp into the video frame, ending encoding, and completing subsequent packaging and other tasks.

[0185] P03 transmission device, the transmission device specifically relates to a network device and a network that can be used for communication. The network device can be connected to other network devices through a base station so as to communicate with the Internet and is used to receive and send data. A specific example of the network may include a wireless network provided by a communication provider. The source of the network includes but is not limited to the Internet, corporate intranet, local area network, mobile communication network and a combination thereof.

[0186] P04 decoding device, the present invention puts the clock synchronization device in the decoding device, and uses a storage device to store the timestamp generated by the clock synchronization device, wherein the storage device is divided into a first storage device and a second storage device, and the storage device can be but not limited to a register, a removable hard disk, a disk and other devices with read, write and storage functions. The device is used to complete the following steps:

[0187] T01: Decode video frames and audio frames and extract encoding timestamps.

[0188] T02: Put the encoded timestamp into a third storage device.

[0189] T03: The clock synchronization device connects to the network to obtain a timestamp and stores it in a fourth storage device. This time is used as the real time when the video frame is played.

[0190] 2. The evaluation module is divided into a calculation device and a judgment device.

[0191] P05 calculation device, used to obtain the encoding timestamp and the real timestamp in the decoding stage. Specifically, the device takes out the encoding timestamp from the third storage device and takes out the real timestamp from the fourth storage device. The calculation device is divided into type A and type B calculation, as shown below:

[0192] Class A calculation: Class A calculation is the calculation method for delay, freeze and I frame loss

[0193] Specifically, the A-type calculation method obtains the current decoding timestamp vdt during decoding i , extract the video encoding timestamp vet i , calculate the video encoding timestamp vet i With the decoded timestamp vdt i The time difference is the live broadcast delay of the video.

[0194] Specifically, the instantaneous delay is calculated as: lat i =vdt i -vet i .

[0195] Specifically, the average delay is calculated as:

[0196] Furthermore, the instantaneous delay and the average delay are subtracted to obtain the difference sequence: seq_t i =lat i -avg_lat.

[0197] Specifically, the computing device generates the average delay of multiple time periods, the instantaneous delay corresponding to each time period, and the frame difference sequence. The average delay of multiple time periods is: avg_lat1, avg_lat2, avg_lat n3 ...avg_lat n , the instantaneous delays of multiple time periods are: lat1, lat2, lat3...lat n , the difference sequence of the frame is: seq_t1, seq_t2, seq_t3...seq_t n , and the average delay avg_lat of multiple time periods.

[0198] Furthermore, online live broadcasts will have a certain delay. Generally speaking, as long as it is within a reasonable range, the quality of the live broadcast is considered to be within an acceptable range.

[0199] Furthermore, freeze and I frame loss are judged in the judgment device based on the statistical data of instantaneous delay and average delay.

[0200] Class B calculation: Class B calculation is the calculation method of audio and video out of sync

[0201] Specifically, get the video encoding timestamp vet during decoding i and audio encoding timestamp aet i , if aet i If it is not 0, the time interval between the video frame and the audio frame is calculated. i=vet i -aet i .

[0202] Furthermore, the actual audio and video time interval aint is obtained. i :aint i =vdt i -adt i .

[0203] Further, calculate the audio and video synchronization difference difft i :diff i =aint i -eint i .

[0204] Further, calculate the average audio and video synchronization difference avg_difft i : Where n means there are n frames.

[0205] Specifically, the calculation device generates audio and video synchronization differences and average video synchronization differences in multiple time periods, and the audio and video synchronization differences in multiple time periods are respectively: difft1, difft2, difft3, ..., difft n The average video synchronization difference is avg_difft i .

[0206] P06 judgment device, in which a series of thresholds are set, various data generated by the calculation device are taken out for analysis and comparison with the thresholds, and finally the quality index results are given.

[0207] Specifically, the average delay of multiple time periods in the computing device, the instantaneous delay corresponding to each time period, and the frame difference sequence are taken out. The average delay of multiple time periods is: avg_lat1, avg_lat2, avg_lat n3 ...avg_lat n , the instantaneous delays of multiple time periods are: lat1, lat2, lat3...lat n , the difference sequence of the frame is: seq_t1, seq_t2, seq_t3...seq_t n , and the average delay avg_lat of multiple time periods.

[0208] Furthermore, a threshold value thr_lat of the average delay is set according to actual conditions, and the judging device compares the threshold value with a plurality of delay data generated by the calculating device, as shown in detail in a below.

[0209] Furthermore, a threshold value thr_seq of the difference sequence is set according to actual conditions, and the judging device compares the instantaneous delay data generated by the calculating device with the threshold value, as shown in b below.

[0210] Specifically, the audio and video synchronization differences and the average video synchronization difference of multiple time periods in the calculation device are taken out, and the audio and video synchronization differences of the multiple time periods are: difft1, difft2, difft3...difft n The average video synchronization difference is avg_difft i .

[0211] Further, the audio and video synchronization threshold thr_vat is set according to actual conditions, and the judgment device compares the audio and video synchronization difference data generated by the calculation device with the threshold, as shown below.

[0212] a. Delay in judging

[0213] The judgment device obtains the average delay avg_lat of multiple time periods, and compares it with the threshold thr_lat. If avg_lat is greater than the threshold thr_lat, it is determined that the video delay is relatively large, and the real delay result avg_lat is output; if avg_lat is less than the threshold thr_lat, it is determined that the video delay is within a normal range, and the real delay result avg_lat is output.

[0214] b. Determine the jam

[0215] like Fig.13 As shown in FIG. 1 , when the delay suddenly increases rapidly, a freeze occurs. The judgment device obtains a difference sequence of seq_t1, seq_t2, seq_t3, ..., seq_t n If the difference sequence is greater than the difference sequence threshold thr_seq, the video frame is determined to be in a stuck state, and the corresponding timestamp and the corresponding instantaneous delay value are output.

[0216] c. Determine I frame loss

[0217] The judgment device obtains the instantaneous delay lat1, lat2, lat3...lat in multiple time periods n If a certain instantaneous delay decreases or is a negative number, it is determined that the video I frame is lost, and the corresponding timestamp and the corresponding instantaneous delay value are output.

[0218] d. Determine if audio and video are out of sync

[0219] The judgment device obtains the audio and video synchronization differences of multiple time periods: difft1, difft2, difft3...difft nIf the audio and video synchronization difference is less than the threshold thr_vat, it is determined that the audio and video in the video quality during the time period are in a synchronized state. If the audio and video synchronization difference is greater than the threshold thr_vat, it is determined that the audio and video in the video quality during the time period are in an unsynchronized state, and the corresponding audio and video timestamp and the corresponding audio and video synchronization difference are output.

[0220] An embodiment of the present invention further provides an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, and when the processor runs the computer program, the steps of the method described in one or more of the above technical solutions are executed.

[0221] An embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. After the computer-executable instructions are executed by a processor, the methods described in one or more of the above-mentioned technical solutions can be implemented.

[0222] The computer storage medium provided in this embodiment may be a non-transient storage medium.

[0223] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0224] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0225] In addition, all functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0226] In some cases, if any two of the above technical features do not conflict with each other, they can be combined into a new method technical solution.

[0227] In some cases, if any two of the above technical features do not conflict with each other, they can be combined into a new equipment technical solution.

[0228] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, etc., various media that can store program codes.

[0229] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimedia data processing method, characterized in that: Applied to the acquisition end, the method includes: Generate a target audio frame and a target video frame according to the multimedia data to be processed; Determining a first timestamp of the target audio frame and a second timestamp of the target video frame; The first timestamp, the second timestamp, the target audio frame, and the target video frame are transmitted to the client together; the first timestamp and the second timestamp are used by the client to determine whether a predetermined abnormality occurs, wherein the predetermined abnormality includes at least one of the following: transmission desynchronization between the target audio frame and the target video frame, transmission abnormality of at least one of the target audio frame and the target video frame, and freeze in the playback of the multimedia data; the first timestamp and the second timestamp are carried in an information frame and transmitted to the client; The transmission desynchronization between the target audio frame and the target video frame is determined by a difference between the first time difference and the second time difference being greater than a first threshold; The first time difference is determined by the difference between the moments indicated by the first timestamp and the second timestamp; the second time difference is determined by the difference between the moments indicated by the third timestamp indicating the receiving moment of the target audio frame and the fourth timestamp indicating the receiving moment of the target video frame.

2. The method according to claim 1, characterized in that The information frame includes: a supplemental enhancement information SEI frame; each group of pictures GOP where the target video frame is located includes one SEI frame.

3. The method according to claim 1, characterized in that The method further comprises: Determining a data type of the multimedia data; If the data type is audio data, determining that the moment when the audio data is acquired is the first moment indicated by the first timestamp, and writing information of the first moment into a first register; the information of the first moment written into the first register is used for generating the first timestamp; If the data type is video data, determine that the moment when the video data is acquired is the second moment indicated by the second timestamp, and write the information of the second moment into the second register; write the information of the first moment into the second register for generating the second timestamp.

4. The method according to claim 3, characterized in that The method further comprises: When determining the first timestamp, reading the information of the first moment from the first register, and resetting a flag bit of the first register, wherein the flag bit of the first register indicates that the first register is set to a preset value; When determining the second timestamp, the second time information is read from the second register.

5. A multimedia data processing method, characterized in that: Applied to a client, the method comprises: Obtain a target audio frame, a target video frame, a first timestamp, and a second timestamp; Determine whether a predetermined abnormality occurs according to the first timestamp and the second timestamp; the predetermined abnormality includes at least one of the following: transmission desynchronization between the target audio frame and the target video frame, transmission abnormality of at least one of the target audio frame and the target video frame, and playback of multimedia data is stuck; the first timestamp and the second timestamp are carried in the information frame and transmitted to the client; The method further comprises: determining a third timestamp indicating a time instant of receiving the target audio frame, and determining a fourth timestamp indicating a time instant of receiving the target video frame; The determining, according to the first timestamp and the second timestamp, whether a predetermined abnormality occurs includes: Determine a first time difference between the first timestamp and the moment indicated by the second timestamp, and determine a second time difference between the moment indicated by the third timestamp and the moment indicated by the fourth timestamp; determining a difference between the first time difference and the second time difference; If the difference is less than or equal to a first threshold, determining that the target audio frame is synchronized with the target video frame; If the difference is greater than the first threshold, it is determined that the target audio frame and the target video frame are out of sync.

6. The method according to claim 5, characterized in that The method further comprises: If the predetermined exception occurs, an exception message is sent to the server.

7. The method according to claim 6, characterized in that The determining whether a predetermined abnormality occurs according to the first timestamp and the second timestamp further includes: Determine the time difference between the second timestamp and the moment indicated by the fourth timestamp as the transmission delay, and / or determine the time difference between the first timestamp and the moment indicated by the third timestamp as the transmission delay; Determine an average delay based on transmission delays of multiple target video frames; Determining whether the predetermined abnormality occurs according to the average delay; The determining, based on the average delay, whether the predetermined abnormality occurs includes at least one of the following: If the average delay is greater than a second threshold, determining that the transmission delay is abnormal; If the difference between the transmission delay and the average delay is greater than a third threshold, it is determined that a freeze occurs in the multimedia data playback.

8. An electronic device, characterized in that: The electronic device comprises: a processor and a memory for storing a computer program that can be run on the processor; wherein, When the processor runs the computer program, the processor executes the steps of the multimedia data processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions; after the computer-executable instructions are executed by the processor, the multimedia data processing method according to any one of claims 1 to 7 can be implemented.

Images