Audio and video processing method and device, electronic equipment and computer program product

Abstract

The invention discloses an audio and video processing method and device, electronic equipment and a computer program product, and the method comprises the steps: determining PTS interval mean values corresponding to received audio and video data streams, the PTS interval mean values being used for representing the mean value of inter-frame PTS interval values of the corresponding data streams; blank PTS intervals corresponding to the audio and video data streams are initialized, the blank PTS intervals are used for recording PTS intervals of blank intervals of the corresponding data streams, and the blank intervals are intervals in which data frames do not exist in the audio and video data streams; according to the PTS interval mean value and the blank PTS interval respectively corresponding to the audio and video data streams, the PTS of the audio and video data streams is processed frame by frame so as to eliminate the influence of the blank interval on the audio and video data streams, and the blank PTS interval is updated in the process of processing the PTS of the audio and video data streams; and determining the playing duration of each data frame according to the PTS of the audio and video data stream, so that the audio and video data stream is kept continuous and synchronous during playing. According to the scheme provided by the invention, the audio and video data stream can still be kept continuous and synchronous during playing under the condition that the received audio and video data stream has data missing.

Description

Technical Field

[0001] The present application belongs to the field of multimedia technology, and in particular relates to an audio and video processing method, an audio and video processing device, an electronic device and a computer program product. Background Art

[0002] During the data transmission of audio and video data streams (including but not limited to data downloading and data exporting), the instability of data transmission or other factors may cause audio frames and / or video frames to be missing in certain time periods. As a result, when the receiver plays the audio and video data stream, the audio and video playback may be discontinuous and out of sync, affecting the user's viewing experience. Summary of the invention

[0003] The present application provides an audio and video processing method, an audio and video processing device, an electronic device and a computer program product, which can keep the audio and video data stream continuous and synchronous during playback when there is data missing in the received audio and video data stream.

[0004] In a first aspect, the present application provides an audio and video processing method, comprising:

[0005] Determine the mean PTS intervals corresponding to the received audio and video data streams, respectively, wherein the audio and video data streams include an audio data stream and a video data stream, and the mean PTS interval is used to represent the mean of the inter-frame PTS interval values ​​of the corresponding data streams;

[0006] Initialize the blank PTS intervals corresponding to the audio and video data streams, respectively, wherein the blank PTS intervals are used to record the PTS intervals of the blank intervals of the corresponding data streams, and the blank intervals are intervals where there are no data frames in the audio and video data streams;

[0007] According to the mean values ​​of the PTS intervals respectively corresponding to the audio and video data streams and the blank PTS intervals respectively corresponding to the audio and video data streams, the PTS of the audio and video data streams are processed frame by frame to eliminate the influence of the blank intervals on the audio and video data streams, wherein the blank PTS intervals respectively corresponding to the audio and video data streams are updated in the process of processing the PTS of the audio and video data streams;

[0008] The playback duration of each data frame is determined according to the PTS of the audio and video data stream, so that the audio and video data stream remains continuous and synchronous during playback.

[0009] In a second aspect, the present application provides an audio and video processing device, comprising:

[0010] A first determination module is used to determine the mean PTS intervals corresponding to the received audio and video data streams, wherein the audio and video data streams include audio data streams and video data streams, and the mean PTS interval is used to represent the mean of the inter-frame PTS interval values ​​of the corresponding data streams;

[0011] An initialization module is used to initialize the blank PTS intervals corresponding to the audio and video data streams, wherein the blank PTS intervals are used to record the PTS intervals of the blank intervals of the corresponding data streams, and the blank intervals are intervals where no data frames exist in the audio and video data streams;

[0012] A processing module, for processing the PTS of the audio and video data stream frame by frame according to the mean values ​​of the PTS intervals respectively corresponding to the audio and video data streams and the blank PTS intervals respectively corresponding to the audio and video data streams, so as to eliminate the influence of the blank intervals on the audio and video data streams, wherein the blank PTS intervals respectively corresponding to the audio and video data streams are updated in the process of processing the PTS of the audio and video data streams;

[0013] The second determination module is used to determine the playing time of each data frame according to the PTS of the audio and video data stream, so that the audio and video data stream remains continuous and synchronous during playback.

[0014] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0016] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect.

[0017] Compared with the prior art, the present application has the following beneficial effects: the present application scheme compensates for the impact of missing data in the audio and video data stream on the playback by adjusting the PTS of the audio and video data stream. Among them, the mean PTS interval is the basis for the receiver to judge whether the audio and video data stream is continuous, which helps the receiver to find the blank interval; and the blank PTS interval is the basis for the receiver to process the PTS of the audio and video data stream, which helps to eliminate the impact of the blank interval on the audio and video data stream. Based on this, the receiver can use the mean PTS interval and the blank PTS interval as the basis for processing the PTS of the audio and video data stream frame by frame, and can update the blank PTS interval during the processing. After the PTS of the audio and video data stream is processed, since the impact of the blank interval on the audio and video data stream has been eliminated, the playback duration of each data frame determined according to the PTS of the audio and video data stream can keep the audio and video data stream continuous and synchronized during playback. It can be understood that the beneficial effects of the second to fifth aspects mentioned above can refer to the relevant description in the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a schematic diagram of the processing links involved in the data transmission process at the receiving end;

[0020] Figure 2 It is a schematic diagram when data is missing;

[0021] Figure 3 This is another schematic diagram when data is missing;

[0022] Figure 4 This is another schematic diagram when data is missing;

[0023] Figure 5 This embodiment of the present application provides Figure 4 A schematic diagram of the expected effect after processing the missing data shown;

[0024] Figure 6 is a schematic diagram of an audio and video processing method provided in an embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of a dual-channel data stream processing flow provided in an embodiment of the present application;

[0026] Figure 8is a schematic diagram of the target end time and the target start time provided in an embodiment of the present application;

[0027] Fig. 9 This embodiment of the present application provides the elimination Figure 8 Schematic diagram of the audio and video data stream after the blank interval shown;

[0028] Fig.10 is a schematic diagram of an audio and video processing device provided in an embodiment of the present application;

[0029] Fig.11 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0031] In order to illustrate the technical solution proposed in this application, a specific embodiment is provided below for illustration.

[0032] The audio and video processing method provided in the embodiment of the present application is applied to the receiving party of the data transmission for the audio and video data stream, and the data transmission includes but is not limited to data downloading and data exporting, etc., which are not limited here. Among them, the receiving party can be any electronic device with data processing capabilities, for example, it can be a smart phone, a tablet computer or a server, etc., and the embodiment of the present application does not limit the specific type of the electronic device.

[0033] In the data transmission process for audio and video data streams, the sender can send the audio and video data streams to the receiver according to the specified start time and end time. It can be understood that the audio and video data streams include audio data streams and video data streams. Figure 1 , Figure 1 The following is a schematic diagram of the processing steps involved in the data transmission process at the receiving end. Figure 1 As shown, the receiving party can process the received audio and video data stream through demultiplexing, decoding, encoding and multiplexing processing links in sequence, and finally encapsulate it into an audio and video file output, and the audio and video file can be played by an electronic device with a playback function. For the audio and video processing method proposed in the embodiment of the present application, it can be executed by the receiving party in any processing link, and the embodiment of the present application does not limit this.

[0034] In actual application scenarios, for the receiver, the audio and video data stream it receives may have the following three data missing situations:

[0035] Case 1: In a certain period of time, there is audio data but no video data, that is, there is data missing in the video data stream, such as Figure 2 As shown in FIG. 1 , V represents the video data stream, and A represents the audio data stream.

[0036] Case 2: In a certain period of time, there is video data but no audio data, that is, there is data missing in the audio data stream, such as Figure 3 As shown in FIG. 1 , V represents the video data stream, and A represents the audio data stream.

[0037] Case 3: In a certain period of time, there is neither video data nor audio data, that is, both the audio data stream and the video data stream are missing data, such as Figure 4 As shown in FIG. 1 , V represents the video data stream, and A represents the audio data stream.

[0038] Among them, situation one and situation two only involve data missing of a single data stream. In this regard, when the receiving party applies the audio and video processing method proposed in the embodiment of the present application, it may consider playing another data stream without data missing within the corresponding time period, thereby achieving the desired effect of freezing the picture but playing the sound normally (for situation one), and the desired effect of playing the picture silently (for situation two).

[0039] Among them, situation three involves data loss in two-way data streams. In this regard, when applying the audio and video processing method proposed in the embodiment of the present application, the receiving party may consider skipping the blank interval. The blank interval is specifically an interval in which there are no data frames in the audio and video data streams (that is, there are data loss in both the audio and video data streams), so as to achieve the desired effect of synchronously skipping the blank interval of the audio and video data streams to maintain normal playback, such as Figure 5 As shown in FIG. 1 , V represents the video data stream, and A represents the audio data stream.

[0040] See also Figure 6 , Figure 6 The implementation process of the audio and video processing method is given as follows:

[0041] Step 601, determining the mean values ​​of the PTS intervals corresponding to the received audio and video data streams.

[0042] As described above, the audio and video data stream includes two data streams, the audio data stream and the video data stream; therefore, the data frames of the audio and video data stream include: the audio frames of the audio data stream, and the video frames of the video data stream. It can be understood that each data frame carries a PTS, but since the time units used by the audio data stream and the video data stream are usually inconsistent (for example, the time unit used by the audio data stream is 8000, and the time unit used by the video data stream is 90000), there is no direct correlation between the PTS of the audio frame and the PTS of the video frame.

[0043] The receiver can first receive the audio and video data streams, and during data reception, there is no need to adjust the PTS of the two data streams, only the data frame information of each data stream needs to be saved; that is, when the receiver receives the audio and video data streams, it does not require that each audio frame in the audio data stream arrive synchronously with each video frame in the video data stream.

[0044] For the received audio and video data streams, the receiver can calculate the mean PTS intervals corresponding to the two data streams respectively, and the mean PTS intervals are used to represent the mean of the inter-frame PTS intervals of the corresponding data streams. The inter-frame PTS intervals mentioned here specifically refer to the difference in PTS between two adjacent data frames in the same data stream received by the receiver.

[0045] In some examples, the receiver can update the mean PTS intervals corresponding to the two data streams in real time according to the currently received data frames during the process of receiving the audio and video data streams, and the updated mean PTS intervals can be overwritten and stored locally at the receiver. Finally, after receiving the last audio frame of the audio data stream, the mean PTS interval corresponding to the audio data stream can be updated; similarly, after receiving the last video frame of the video data stream, the mean PTS interval corresponding to the video data stream can be updated. The mean PTS intervals to be determined in this step refer to the mean PTS intervals corresponding to the audio data stream and the mean PTS intervals corresponding to the video data stream.

[0046] In this way, the receiver can always obtain the latest PTS interval average based on the latest situation of the currently received audio and video data stream. This helps to restore the reception process based on the PTS interval averages corresponding to the two data streams before the interruption when the reception is interrupted. This process will be described in detail later and will not be repeated here.

[0047] In other examples, the receiver may also perform calculations based on all data frames of the audio and video data streams received this time after receiving the audio and video data streams, so as to obtain the average PTS intervals corresponding to the audio and video data streams respectively.

[0048] In this way, the receiver only needs to calculate the PTS interval mean once for both the audio data stream and the video data stream, which can greatly reduce the amount of calculation. However, since this method cannot obtain the real-time PTS interval mean corresponding to the two data streams during the receiving process, it cannot cope with the situation where the reception is interrupted.

[0049] It should be noted that no matter which method is used to determine the mean PTS interval, the receiver can perform a screening operation, specifically to remove the inter-frame PTS intervals of discontinuous adjacent frames in the data stream. It can be understood that under normal circumstances, the arrangement between the data frames of the data stream is relatively tight, so that a continuous effect can be formed during playback, that is, the inter-frame PTS intervals between continuous adjacent frames will not be too large; however, due to factors such as the instability of data transmission, some data frames may be lost between two data frames that are adjacent in time sequence received by the receiver, resulting in the two data frames that are adjacent in time sequence being substantially discontinuous and showing the characteristics of an excessively large inter-frame PTS interval. Based on this feature, the inter-frame PTS intervals of discontinuous adjacent frames (that is, an excessively large inter-frame PTS interval) can be removed from the calculation, thereby ensuring the accuracy of the calculation of the mean PTS interval.

[0050] Step 602, initialize the blank PTS intervals corresponding to the audio and video data streams respectively.

[0051] Based on the expected effects of the above-mentioned cases 1, 2 and 3, it can be known that when there is data missing in a single data stream, the audio and video data streams can maintain the original corresponding relationship, that is, the receiver does not need to adjust the PTS of the audio and video data streams; only when there is data missing in both data streams, the receiver needs to adjust the PTS of the audio and video data streams after the blank interval, so that the blank interval can be skipped during playback. Based on this, the receiver can first create blank PTS intervals corresponding to the audio and video data streams, respectively, and initialize the blank PTS intervals to 0. It can be understood that the blank PTS interval belongs to a global variable, which can be used to record the PTS interval of the blank interval of the corresponding data stream.

[0052] Step 603, based on the mean values ​​of the PTS intervals corresponding to the audio and video data streams and the blank PTS intervals corresponding to the audio and video data streams, the PTS of the audio and video data streams are processed frame by frame to eliminate the influence of the blank intervals on the audio and video data streams.

[0053] The receiver can process the PTS of the audio and video data stream frame by frame until the PTS of all data frames in the audio and video data stream are processed. Based on the previous description, it can be seen that there is data missing between discontinuous adjacent frames, and the inter-frame PTS intervals of discontinuous adjacent frames are usually large; based on this, the receiver can use the mean of the PTS intervals corresponding to the audio and video data streams as the basis for judging whether there are discontinuous adjacent frames in the audio and video data streams, thereby accurately identifying the blank intervals. Since the existence of the blank interval will affect the PTS of the dual-channel video stream after the blank interval, and the time units used by the audio and video data streams are usually inconsistent, the receiver can update the blank PTS intervals corresponding to the audio and video data streams in time after determining the blank interval, thereby ensuring the accuracy of the PTS adjustment. That is, the blank PTS intervals corresponding to the audio and video data streams can be updated in the process of the receiver processing the PTS of the audio and video data streams.

[0054] Step 604, determining the playing duration of each data frame according to the PTS of the audio and video data stream, so that the audio and video data stream remains continuous and synchronous during playing.

[0055] The receiving party can calculate the playing time of each data frame according to the PTS after processing each data frame of the audio and video data stream, and complete the index encapsulation of the audio and video file, and the audio and video file can be played by an electronic device with a playback function. Taking any video frame of the video data stream as an example, the playing time of the video frame can be calculated according to the PTS after processing the video frame, and the PTS after processing the next frame of the video frame. It can be understood that when the video frame and the next frame of the video frame are discontinuous adjacent frames (that is, when there is data missing between the video frame and the next frame of the video frame), the display of the video frame can be extended through this step, so that the picture is frozen and played. Similarly, taking any audio frame of the audio data stream as an example, the playing time of the audio frame can be calculated according to the PTS after processing the audio frame, and the PTS after processing the next frame of the audio frame. It can be understood that when the audio frame and the frame following the audio frame are discontinuous adjacent frames (that is, there is data missing between the audio frame and the frame following the audio frame), this step can extend the playback of the audio frame, that is, extend the end time of the audio frame, so that the picture is played silently. In addition, since the PTS after each data frame processing eliminates the influence of the blank interval, the audio and video data stream can be played as follows Figure 5 That way the blank intervals are skipped directly, thus keeping the playback continuous and synchronized.

[0056] In some embodiments, when the receiver processes the PTS of the audio and video data stream frame by frame, step 603 can be specifically divided into two different application scenarios and processing methods, which are described in detail as follows:

[0057] Application scenario 1: Currently, there are unprocessed frames in both the audio data stream and the video data stream. That is, in this application scenario, the receiver has not yet completed the processing of any of the audio and video data streams. In this regard, the receiver can trigger a dual-channel data stream processing flow, which can adopt a strategy in which audio frames and video frames refer to each other and are mutually standardized, and process the PTS of the audio data stream and the video data stream frame by frame according to the average PTS interval corresponding to the audio data stream and the video data stream, and the blank PTS interval corresponding to the audio data stream and the video data stream.

[0058] Application scenario two: Currently, only one of the audio data stream and the video data stream has unprocessed frames. That is, in this application scenario, the receiver has completed the processing of one of the data streams, and only the other data stream has not been processed. In this regard, the receiver can trigger a single-channel data stream processing flow, which can adopt a direct splicing strategy to process the PTS of the unprocessed data stream frame by frame according to the average PTS interval corresponding to the unprocessed data stream and the blank PTS interval corresponding to the unprocessed data stream. Among them, the unprocessed data stream mentioned here refers to: the data stream where the unprocessed frame is located (that is, the data stream that has not yet been processed). That is, if only the audio data stream currently has unprocessed frames, the unprocessed data stream is the audio data stream; similarly, if only the video data stream currently has unprocessed frames, the unprocessed data stream is the video data stream.

[0059] To facilitate understanding of this solution, the following describes the dual-channel data flow processing process. Figure 7 , Figure 7 The schematic diagram of the dual-channel data stream processing flow is given as follows:

[0060] Step 701: determine a frame to be processed and a reference frame in the next unprocessed frame of the audio data stream and the next unprocessed frame of the video data stream.

[0061] As mentioned above, the dual-channel data stream processing process adopts a strategy in which audio frames and video frames are referenced and standardized to each other. Based on this, in the dual-channel data stream processing process, for the next unprocessed frame of the audio data stream and the next unprocessed frame of the video data stream, the receiver can first determine the frame to be processed and the reference frame. It can be understood that the frame to be processed refers to the frame whose PTS is prepared to be processed this time; the reference frame refers to the frame used as a reference when processing the PTS of the frame to be processed. Considering that the receiver processes each data frame of the audio and video data stream frame by frame based on the time sequence, the frame to be processed and the reference frame can be determined according to the order relationship of the next unprocessed frame of the audio data stream and the next unprocessed frame of the video data stream in the time dimension of the audio and video data stream, wherein the frame to be processed should be earlier than the reference frame in this time dimension. At the same time, for the convenience of description, the data stream where the frame to be processed is located is recorded as the data stream to be processed, and the data stream where the reference frame is located is recorded as the reference data stream.

[0062] In some examples, the receiver may determine the frame to be processed and the comparison frame in the following manner:

[0063] A1. Determine the start time of the next unprocessed frame of the audio data stream based on the PTS of the next unprocessed frame of the audio data stream and the time unit adopted by the audio data stream.

[0064] A2. Determine the start time of the next unprocessed frame of the video data stream based on the PTS of the next unprocessed frame of the video data stream and the time unit adopted by the video data stream.

[0065] A3. Among the next unprocessed frame of the audio data stream and the next unprocessed frame of the video data stream, an unprocessed frame with a relatively small start time (i.e., a relatively early unprocessed frame) is determined as a to-be-processed frame, and an unprocessed frame with a relatively large start time (i.e., a relatively late unprocessed frame) is determined as a control frame.

[0066] Specifically, the receiver can determine the ratio of the PTS of the next unprocessed frame of the audio data stream to the time unit used by the audio data stream as the start time of the next unprocessed frame of the audio data stream. Similarly, the ratio of the PTS of the next unprocessed frame of the video data stream to the time unit used by the video data stream can also be determined as the start time of the next unprocessed frame of the video data stream. It can be understood that the calculated start time can be used to indicate the number of seconds of data in the corresponding data stream for the corresponding unprocessed frame. By comparing the start time, the receiver can determine the frame to be processed and the reference frame.

[0067] Step 702: When the frame to be processed is not the first frame of the data stream to be processed, determine whether to update the blank PTS intervals corresponding to the audio data stream and the video data stream respectively according to a preset update condition.

[0068] When the frame to be processed is the first frame of the data stream to be processed, since the first data frame must not have a previous data frame to be compared with it, that is, there will be no blank interval before the first data frame, the receiver does not need to adjust its PTS, and thus there is no need to update the blank PTS interval at this time.

[0069] In the case where the frame to be processed is not the first frame of the data stream to be processed, it can be known from the above description that the receiver will update the blank PTS intervals corresponding to the audio data stream and the video data stream respectively only when there is a blank interval. Based on this, the receiver can pre-set the update condition to determine whether there is a blank interval, so as to determine whether to update the blank PTS interval.

[0070] In some examples, when the frame to be processed is not the first frame of the data stream to be processed, the receiver may determine whether to update the blank PTS interval in the following manner:

[0071] B1. Calculate the first PTS interval and the second PTS interval.

[0072] According to the definition of blank interval in the previous text, a blank interval is considered to exist only when there is data missing between the frame to be processed and the frame before the frame to be processed, and there is data missing between the reference frame and the frame before the reference frame. It should be noted that the frame before the frame to be processed specifically refers to the frame before the frame to be processed in the data stream to be processed (that is, the latest processed frame in the data stream to be processed); similarly, the frame before the reference frame specifically refers to the frame before the reference frame in the reference data stream (that is, the latest processed frame in the reference data stream).

[0073] Based on this, in order to make further judgment on the existence or non-existence of the blank interval, the receiver can calculate the first PTS interval and the second PTS interval, wherein the first PTS interval is: the PTS interval between the frame to be processed and the previous frame to be processed (that is, the difference between the PTS of the frame to be processed and the PTS of the previous frame to be processed), and the second PTS interval is: the PTS interval between the control frame and the previous frame of the control frame (that is, the difference between the PTS of the control frame and the PTS of the previous frame of the control frame).

[0074] B2. Determine whether the first PTS interval is smaller than a preset first PTS interval threshold, and whether the second PTS interval is smaller than a preset second PTS interval threshold.

[0075] To avoid misjudgment, the embodiment of the present application proposes to use the first PTS interval threshold as a criterion for judging whether adjacent frames of the data stream to be processed are continuous, and to use the second PTS interval threshold as a criterion for judging whether adjacent frames of the control data stream are continuous. Among them, the first PTS interval threshold can be determined based on the mean PTS interval corresponding to the data stream to be processed. Similarly, the second PTS interval threshold can be determined based on the mean PTS interval corresponding to the control data stream. For example, the first PTS interval threshold can be a specified multiple of the mean PTS interval corresponding to the data stream to be processed, and the second PTS interval threshold can be a specified multiple of the mean PTS interval corresponding to the control data stream, and the specified multiple is greater than 1.

[0076] B3. When the first PTS interval is not less than the first PTS interval threshold and the second PTS interval is not less than the second PTS interval threshold, determine to update the blank PTS intervals corresponding to the audio data stream and the video data stream respectively.

[0077] When the first PTS interval is not less than the first PTS interval threshold, it means that the time difference between the frame to be processed and the previous frame to be processed is large, that is, there is a high probability of data missing between the frame to be processed and the previous frame to be processed. Similarly, when the second PTS interval is not less than the second PTS interval threshold, it means that the time difference between the control frame and the previous frame to be processed is large, that is, there is a high probability of data missing between the control frame and the previous frame to be processed. Based on this, when the first PTS interval is not less than the first PTS interval threshold, and the second PTS interval is not less than the second PTS interval threshold, the receiver can confirm that a blank interval has been identified. At this time, it is necessary to update the blank PTS intervals corresponding to the audio data stream and the video data stream respectively.

[0078] In fact, if only the first PTS interval is not less than the first PTS interval threshold, it can be known that there is data missing only between the to-be-processed frame and the frame preceding the to-be-processed frame; similarly, if only the second PTS interval is not less than the second PTS interval threshold, it can be known that there is data missing only between the comparison frame and the frame preceding the comparison frame. Figure 2 , Figure 3 As can be seen from the foregoing description, when data is missing in a single data stream, it is not necessary to update the blank PTS intervals corresponding to the audio data stream and the video data stream respectively.

[0079] Step 703, when it is determined that the blank PTS intervals corresponding to the audio data stream and the video data stream are to be updated, the blank PTS intervals corresponding to the audio data stream and the video data stream are updated according to the average values ​​of the PTS intervals corresponding to the frame to be processed, the previous frame of the frame to be processed, the reference frame, the previous frame of the reference frame, the audio data stream, and the video data stream.

[0080] When a blank interval is identified, that is, the blank PTS interval needs to be updated, the receiver can comprehensively consider the frame to be processed, the previous frame of the frame to be processed, the reference frame and the previous frame of the reference frame, so as to relatively accurately determine the duration of the blank interval, and thus the blank PTS intervals corresponding to the updated audio and video data streams can relatively accurately reflect the impact of the blank interval on the PTS of subsequent data frames.

[0081] In some examples, the receiver may update the blank PTS interval as follows:

[0082] C1. Determine the end time of the previous frame of the frame to be processed, the end time of the previous frame of the reference frame, the start time of the frame to be processed, and the start time of the reference frame.

[0083] Among them, the end time of the previous frame of the frame to be processed can be calculated in the following way: after adding the PTS of the previous frame to the average PTS interval corresponding to the data stream to be processed, the result is approximately equal to the PTS corresponding to the end time of the previous frame; based on this, the ratio of the result obtained and the time unit used by the data stream to be processed can be determined as the end time of the previous frame.

[0084] Similarly, the end time of the previous frame of the reference frame can be calculated in the following way: after adding the PTS of the previous frame to the average PTS interval corresponding to the reference data stream, the result is approximately equal to the PTS corresponding to the end time of the previous frame; based on this, the ratio of the result to the time unit used by the reference data stream can be determined as the end time of the previous frame.

[0085] The start time of the frame to be processed is specifically the ratio of the PTS of the frame to be processed to the time unit adopted by the data stream to be processed.

[0086] Similarly, the start time of the comparison frame is specifically the ratio of the PTS of the comparison frame to the time unit adopted by the comparison data stream.

[0087] C2. Among the end time of the previous frame of the frame to be processed and the end time of the previous frame of the reference frame, the relatively larger end time is used as the target end time.

[0088] C3. Among the start time of the frame to be processed and the start time of the reference frame, the relatively smaller start time is used as the target start time.

[0089] See also Figure 8 , Figure 8 The target end time and target start time are shown in Figure 1. V represents the video data stream, A represents the audio data stream, PreEndTimeMax represents the target end time, and CurStartTimeMin represents the target start time. Figure 8 As shown, the target end time refers to the later end time of the data frames of the two data streams before the blank interval, which actually corresponds to the start time of the blank interval; the target start time refers to the earlier start time of the data frames of the two data streams after the blank interval, which actually corresponds to the end time of the blank interval.

[0090] C4. Calculate the time interval of the blank interval based on the target end time and the target start time.

[0091] like Figure 8 As shown in the figure, the target start time minus the target end time is the time interval of the blank interval, which can be specifically expressed by the following formula:

[0092] TimeInterval=CurStartTimeMin–PreEndTimeMax

[0093] Among them, TimeInterval is the time interval of the blank interval identified this time, and CurStartTimeMin and PreEndTimeMax have been described in the previous article and will not be repeated here.

[0094] C5. Update the blank PTS intervals corresponding to the audio and video data streams respectively according to the time interval of the blank interval.

[0095] Since both audio and video data streams need to skip blank intervals, the blank PTS intervals corresponding to the audio and video data streams need to be updated. Specifically, for the audio data stream, it can be based on the current corresponding blank PTS interval (that is, the blank PTS interval that has not been updated), plus the PTS increase value corresponding to the blank interval identified this time in the audio data stream, so as to obtain the updated blank PTS interval corresponding to the audio data stream. Similarly, for the video data stream, it can be based on the current corresponding blank PTS interval (that is, the blank PTS interval that has not been updated), plus the PTS increase value corresponding to the blank interval identified this time in the video data stream, so as to obtain the updated blank PTS interval corresponding to the video data stream. As an example only, in the code, the above update of the blank PTS interval can be specifically expressed as:

[0096] interval A +=TimeInterval*TimeScale A

[0097] interval V +=TimeInterval*TimeScale V

[0098] Among them, the subscript A represents the audio data stream, and the subscript V represents the video data stream; interval A Indicates the blank PTS interval corresponding to the audio data stream, TimeScale A The time unit used to represent the audio data stream; interval V Indicates the blank PTS interval corresponding to the video data stream, TimeScale V It is the time unit used to represent the video data stream; TimeInterval represents the time interval of the blank interval identified this time.

[0099] Step 704: Update the PTS of the frame to be processed according to the blank PTS interval corresponding to the data stream to be processed.

[0100] Since the receiver performs frame-by-frame adjustment, the receiver only updates the PTS of the frame to be processed. As an example, in the code, the above update of the PTS of the frame to be processed can be specifically expressed as:

[0101] curPTS=curPTS–interval cur

[0102] Among them, curPTS represents the PTS of the frame to be processed, interval cur Indicates the blank PTS interval corresponding to the data stream to be processed. That is, the receiver can subtract the blank PTS interval corresponding to the data stream to be processed from the PTS of the frame to be processed to realize the processing of the PTS of the frame to be processed. It should be noted that the blank PTS intervals corresponding to the audio and video data streams are global variables. Therefore, if the blank PTS interval is updated through step 703, the blank PTS interval corresponding to the data stream to be processed in this step is actually the updated blank PTS interval corresponding to the data stream to be processed.

[0103] Understandably, for Figure 8 The blank interval shown in the figure can be obtained by adjusting the PTS of the audio and video data stream after the blank interval frame by frame through the dual-channel data stream processing process proposed in the previous article. Fig. 9 . Where V represents the video data stream and A represents the audio data stream. That is, Fig. 9 Given the elimination Figure 8 The following is a schematic diagram of the audio and video data stream obtained after the blank interval shown. Fig. 9 The gap between A1 and A2 in V1 can be played silently by processing the step 604 mentioned above. Fig. 9 The gap between V1 and V2 in V1 can be played by freezing the last video frame in V1 through the processing of step 604 mentioned above.

[0104] In some embodiments, when the receiver updates the PTS of the frame to be processed, it may also synchronously update the decoding timestamp (DTS) of the frame to be processed. As an example only, the updated DTS of the frame to be processed may be equal to the updated PTS of the frame to be processed.

[0105] It can be understood that after the PTS of the frame to be processed is updated, the receiver has completed the processing of the frame to be processed. In this case, the receiver can reconfirm whether the current application scenario is the application scenario 1 or the application scenario 2 proposed above (because the frame to be processed may be the last frame of the data stream to be processed), so as to select the appropriate processing flow in the dual-channel data stream processing flow and the single-channel data stream processing flow to continue processing.

[0106] To facilitate understanding of this solution, the single-channel data stream processing flow is described below, wherein the processing object of the single-channel data stream processing flow is the next unprocessed frame of the unprocessed data stream, as described in detail as follows:

[0107] First, the PTS interval between the unprocessed frame and the previous frame of the unprocessed frame is calculated, and it is determined whether the PTS interval is less than the PTS interval threshold corresponding to the unprocessed data stream. The PTS interval threshold can be determined based on the mean PTS interval corresponding to the unprocessed data stream, for example, the PTS interval threshold can be a specified multiple of the mean PTS interval corresponding to the unprocessed data stream, and the specified multiple is greater than 1.

[0108] Subsequently, when the PTS interval is less than the PTS interval threshold corresponding to the unprocessed data stream, the blank PTS interval corresponding to the unprocessed data stream is not updated. On the contrary, when the PTS interval is not less than the PTS interval threshold corresponding to the unprocessed data stream, it can be confirmed that there is a blank interval before the unprocessed frame (the data frame of the other data stream has been processed, so as long as there is data missing before the unprocessed frame, there must be a blank interval). At this time, the blank PTS interval corresponding to the unprocessed data stream can be updated, which can be expressed by the formula: interval = interval + CurPTS-(PrePTS + AvrPTS). Among them, interval represents the blank PTS interval corresponding to the unprocessed data stream, CurPTS represents the PTS of the unprocessed frame, PrePTS represents the PTS of the previous frame of the unprocessed frame, and AvrPTS represents the average PTS interval corresponding to the unprocessed data stream.

[0109] Finally, regardless of whether the blank PTS interval corresponding to the unprocessed data stream is updated, the receiver can update the PTS of the unprocessed frame based on CurPTS=CurPTS-interval.

[0110] It is understood that after the PTS of the unprocessed frame is updated, the receiver has completed the processing of the unprocessed frame. In this case, if there are still data frames that have not been processed in the unprocessed data stream, the next unprocessed frame can be processed again through the above-mentioned single-channel data stream processing flow, which will not be described in detail here.

[0111] In some embodiments, if the receiver updates the PTS interval mean values ​​corresponding to the two data streams in real time according to the currently received data frames during the process of receiving the audio and video data streams, the receiver can always obtain the latest PTS interval mean values ​​corresponding to the audio and video data streams during the process. Based on this, the embodiment of the present application also proposes relevant processing when resuming reception after the reception is interrupted. The following is a brief description of the problems caused by the interruption of reception:

[0112] As described above, when the receiver receives the audio and video data streams, it is not strictly required that the two data streams arrive synchronously. Therefore, when the data transmission is interrupted due to instability of data transmission or other factors, the lengths of the two data streams received at the interruption point may not be equal. For example, when the interruption occurs, the receiver may have received 5 minutes of video data and 5 and a half minutes of audio data.

[0113] When restoring the interruption, in order to ensure that the data frames near the interruption point are not lost, the sender often transmits some extra data. Assuming that the sender starts to resend data from 4 minutes and 55 seconds, then according to the data transmission mechanism, the sender will specifically resend the audio and video data stream from the position closest to the time point of 4 minutes and 55 seconds, and the sender does not guarantee that the first audio frame and the first video frame resent are completely aligned. In this way, for the receiver, after resuming reception, the audio and video data streams obtained will have a certain degree of data repetition, and the repetition lengths are not necessarily equal. In addition, after the receiver resumes reception, for any data stream, the PTS of the data frame in the data stream will mutate, and the PTS of the mutated data frame has no correlation with the PTS of the data frame before the interruption.

[0114] In this regard, the reception before and after the interruption is considered to be two different receptions. The embodiment of the present application mainly focuses on the following two core ideas and proposes relevant processing when resuming reception:

[0115] 1. In different receptions, the UTC time of the video data corresponding to the same time point in the same audio and video data stream is always the same.

[0116] 2. In the same reception, for the same time point, the time unit adopted by the PTS of the audio frame / audio data stream at that time point = the time unit adopted by the PTS of the video frame / video data stream at that time point.

[0117] Based on the above core ideas, the relevant processing when resuming reception proposed in the embodiment of the present application is described in detail below:

[0118] D1. During the process of receiving the audio and video data stream, if an interruption occurs, the time parameters of the audio and video data stream before the interruption are determined.

[0119] Based on the core idea proposed in the previous article, it can be known that the UTC time of the video frame is the basis for associating different receptions; however, in the video data stream, only the video I frame carries the UTC time (that is, not all video frames carry the UTC time). Based on this, the receiver can first determine the target interruption video frame and the target interruption audio frame after the interruption occurs. Among them, the target interruption video frame refers to: the last video frame with UTC time received before this interruption, and the target interruption audio frame refers to: the last audio frame received before this interruption.

[0120] The receiver can determine the time parameters according to the target interruption video frame and the target interruption audio frame, and the time parameters include: the PTS of the target interruption video frame, the PTS of the target interruption audio frame, the time when the video frame reception before the interruption is completed, and the time when the audio frame reception before the interruption is completed.

[0121] The video frame receiving completion time may be determined according to the UTC time of the target interrupted video frame. As an example only, the video frame receiving completion time may be directly equivalent to the UTC time of the target interrupted video frame.

[0122] Among them, the audio frame reception completion time is determined according to the time unit respectively adopted by the audio and video data streams, the PTS of the target interrupt audio frame, the UTC time and PTS of the target interrupt video frame. Specifically, the receiver can first calculate the time difference between the target interrupt audio frame and the target interrupt video frame according to the time unit respectively adopted by the audio and video data streams, the PTS of the target interrupt audio frame and the PTS of the target interrupt video frame, which can be specifically expressed as: lastAudioPTS / audioTimeScale-lastVideoPTS / videoTimeScale. It can be understood that the difference may be a positive value (indicating that the target interrupt audio frame is after the target interrupt video frame) or a negative value (indicating that the target interrupt audio frame is before the target interrupt video frame). Then, the receiver can add the time difference to the video frame reception completion time (that is, the UTC time of the target interrupt video frame) to obtain the audio frame reception completion time.

[0123] D2. In the case of recovery reception, determine the target recovery video frame and the target recovery audio frame.

[0124] The target recovery video frame is the first valid video frame received after the recovery reception, and the target recovery audio frame is the first valid audio frame received after the target recovery video frame. It can be understood that since only video frames carry UTC time, the actual time of the audio frame cannot be known before the receiver determines the target recovery video frame, resulting in the receiver being unable to determine whether the audio frame is received repeatedly; based on this, the receiver does not process the audio frame before receiving the target recovery video frame.

[0125] In some examples, after resuming reception, the receiver may determine the target restored video frame in the following manner:

[0126] When the current video frame is received, determine whether the current video frame carries the UTC time. It can be understood that when the current video frame does not carry the UTC time, it indicates that the current video frame is a video P frame, and the receiver can discard the current video frame. On the contrary, when the current video frame carries the UTC time, it indicates that the current video frame is a video I frame, and the receiver can further compare the UTC time of the current video frame with the determined video frame reception completion time to determine whether it is repeated with the video frame received before the interruption. When the UTC time of the current video frame is greater than the video frame reception completion time, it can be known that the current video frame is the first video I frame that is not repeatedly received after the recovery reception, and thus the current video frame can be determined as the target recovery video frame. On the contrary, when the UTC time of the current video frame is not greater than the video frame reception completion time, it can be considered that the current video frame is repeated with the video frame received before the interruption, and thus the receiver can discard the current video frame and wait for the next video frame to be received. The above process is repeated until the target recovery video frame is determined.

[0127] That is, for the video frame obtained after the recovery reception, it needs to meet the following two conditions to be determined as the target recovery video frame: Condition 1, carrying UTC time; Condition 2: the UTC time is greater than the completion time of the video frame reception.

[0128] In some examples, after resuming reception, the receiver may determine the target restored audio frame in the following manner:

[0129] After determining the target restored video frame, once the current audio frame is received, the receiver can first determine the start time of the current audio frame. The start time of the current audio frame can be determined by the following formula: curAudioTime = firstUTC + PTS / audioTimeScale - firstPTS / videoTimeScale. Wherein, curAudioTime represents the start time of the current audio frame, firstUTC represents the UTC time of the target restored video frame, PTS represents the PTS of the current audio frame, audioTimeScale represents the time unit used by the audio data stream, firstPTS represents the PTS of the target restored video frame, and videoTimeScale represents the time unit used by the video data stream. Subsequently, the receiver can compare the start time of the current audio frame with the audio frame reception completion time to determine whether it is repeated with the audio frame received before the interruption. In the case that the start time of the current audio frame is greater than the audio frame reception completion time, it can be known that the current audio frame is the first non-repeatedly received audio frame after the target restored video frame, and thus the current audio frame can be determined as the target restored audio frame. On the contrary, if the start time of the current audio frame is not greater than the audio frame reception completion time, it can be considered that the current audio frame is repeated with the audio frame received before the interruption, so the receiver can discard the current audio frame and wait for the next audio frame to be received. The above process is repeated until the target audio frame is determined.

[0130] That is, for the audio frame obtained after the recovery reception, it needs to meet the following two conditions to be determined as the target recovery video frame: Condition 1, received after the target recovery video frame; Condition 2: the start time of the audio frame is greater than the completion time of the audio frame reception.

[0131] D3. According to the target restored video frame and time parameters, the PTS of the target restored video frame and subsequent video frames are adjusted.

[0132] Among them, the subsequent video frame is: the video frame received after the target recovery video frame. Specifically, according to the PTS of the target recovery video frame, the PTS of the target interrupted video frame and the average of the PTS intervals corresponding to the video data stream before the interruption, the receiver can calculate the video stream interruption PTS interval, which is used to indicate the impact of the interruption on the PTS of the video frame (equivalent to the sudden change of the PTS of the video data stream caused by the interruption). As an example only, the calculation formula of the video frame PTS adjustment difference can be specifically expressed as:

[0133] VideoRestartInterval=firstVideoPTS–(lastVideoPTS+AvrVideoPTS)

[0134] Among them, VideoRestartInterval represents the video frame PTS adjustment difference, firstVideoPTS represents the PTS of the target restored video frame, lastVideoPTS represents the PTS of the target interrupted video frame, and AvrVideoPTS represents the average PTS interval corresponding to the video data stream before the interruption.

[0135] For the target restored video frame and each subsequent video frame after this interruption, the receiver can subtract the PTS adjustment difference of the video frame from its PTS, thereby associating the two different receptions before and after the interruption, so that the PTS of each video frame in the video data stream before and after the interruption is no longer unrelated due to the interruption, but keeps increasing continuously.

[0136] D4. According to the target restored audio frame and time parameters, the PTS of the target restored audio frame and subsequent audio frames are adjusted.

[0137] Among them, the subsequent audio frame is: the audio frame received after the target recovery audio frame. Specifically, according to the PTS of the target recovery audio frame, the PTS of the target interrupted audio frame and the average of the PTS intervals corresponding to the audio data stream before the interruption, the receiver can calculate the audio stream interruption PTS interval, which is used to indicate the impact of the interruption on the PTS of the audio frame (equivalent to the mutation value of the PTS of the audio data stream caused by the interruption). As an example only, the calculation formula of the audio frame PTS adjustment difference can be specifically expressed as:

[0138] AudioRestartInterval=firstAudioPTS–(lastAudioPTS+AvrAudioPTS)

[0139] Among them, AudioRestartInterval represents the audio frame PTS adjustment difference, firstAudioPTS represents the PTS of the target restored audio frame, lastAudioPTS represents the PTS of the target interrupted audio frame, and AvrAudioPTS represents the average PTS interval corresponding to the audio data stream before the interruption.

[0140] Similar to subsequent video frames, for the target recovery audio frame and each subsequent audio frame of this interruption, the receiver can subtract the PTS adjustment difference of the audio frame from their PTS, thereby associating the two different receptions before and after the interruption, so that the PTS of each audio frame in the audio data stream before and after the interruption are no longer unrelated due to the interruption, but keep increasing continuously.

[0141] It should be noted that if multiple interruptions occur during the data transmission process, steps D1 to D4 may be used to process each interruption, which will not be described in detail here.

[0142] It can be understood that through steps D1-D4, the receiver can make preliminary PTS adjustments to the audio and video data stream based on possible interruptions during the process of receiving the audio and video data stream, thereby ensuring that the PTS of the audio frame and the PTS of the video frame finally received will not change suddenly due to the interruption, and will not be received repeatedly due to the interruption. Based on this, through the steps 601-604 proposed above, the receiver can make further PTS adjustments to the audio and video data stream based on possible data missing conditions after receiving the audio and video data stream, so that the audio and video data stream remains continuous and synchronized during playback.

[0143] As can be seen from the above, the embodiment of the present application compensates for the impact of missing data in the audio and video data stream on the playback by adjusting the PTS of the audio and video data stream. Among them, the mean PTS interval is the basis for the receiver to determine whether the audio and video data stream is continuous, which helps the receiver to find the blank interval; and the blank PTS interval is the basis for the receiver to process the PTS of the audio and video data stream, which helps to eliminate the impact of the blank interval on the audio and video data stream. Based on this, the receiver can use the mean PTS interval and the blank PTS interval as the basis for processing the PTS of the audio and video data stream frame by frame, and can update the blank PTS interval during the processing. After the PTS of the audio and video data stream is processed, since the impact of the blank interval on the audio and video data stream has been eliminated, the playback duration of each data frame determined according to the PTS of the audio and video data stream can keep the audio and video data stream continuous and synchronized during playback.

[0144] Corresponding to the audio and video processing method provided above, an embodiment of the present application also provides an audio and video processing device.

[0145] like Fig.10 As shown, the audio and video processing device 10 includes:

[0146] The first determination module 1001 is used to determine the mean PTS intervals corresponding to the received audio and video data streams, wherein the audio and video data streams include audio data streams and video data streams, and the mean PTS interval is used to represent the mean of the inter-frame PTS interval values ​​of the corresponding data streams;

[0147] Initialization module 1002, used to initialize the blank PTS intervals corresponding to the audio and video data streams, wherein the blank PTS intervals are used to record the PTS intervals of the blank intervals of the corresponding data streams, and the blank intervals are intervals where no data frames exist in the audio and video data streams;

[0148] The processing module 1003 is used to process the PTS of the audio and video data stream frame by frame according to the mean values ​​of the PTS intervals respectively corresponding to the audio and video data streams and the blank PTS intervals respectively corresponding to the audio and video data streams, so as to eliminate the influence of the blank intervals on the audio and video data streams, wherein the blank PTS intervals respectively corresponding to the audio and video data streams are updated in the process of processing the PTS of the audio and video data streams;

[0149] The second determination module 1004 is used to determine the playing time of each data frame according to the PTS of the audio and video data stream, so that the audio and video data stream remains continuous and synchronous during playing.

[0150] In some embodiments, the processing module 1003 includes:

[0151] A first processing submodule is used to trigger a dual-channel data stream processing flow when there are unprocessed frames in both the audio data stream and the video data stream, and the dual-channel data stream processing flow is used to process the PTS of the audio data stream and the video data stream frame by frame according to the average values ​​of the PTS intervals respectively corresponding to the audio data stream and the video data stream, and the blank PTS intervals respectively corresponding to the audio data stream and the video data stream;

[0152] The second processing submodule is used to trigger a single-channel data stream processing flow when there is only one data stream with unprocessed frames among the audio data stream and the video data stream. The single-channel data stream processing flow is used to process the PTS of the unprocessed data stream frame by frame according to the average PTS interval corresponding to the unprocessed data stream and the blank PTS interval corresponding to the unprocessed data stream, wherein the unprocessed data stream is: the data stream where the unprocessed frames are located.

[0153] In some embodiments, the first processing submodule includes:

[0154] A first determining unit, configured to determine a frame to be processed and a reference frame in a next unprocessed frame of the audio data stream and a next unprocessed frame of the video data stream;

[0155] The second determining unit is used to determine whether to update the blank PTS intervals respectively corresponding to the audio data stream and the video data stream according to a preset update condition when the frame to be processed is not the first frame of the data stream to be processed, wherein the data stream to be processed is: the data stream where the frame to be processed is located;

[0156] A first updating unit is used to update the blank PTS intervals corresponding to the audio data stream and the video data stream respectively according to the mean values ​​of the PTS intervals corresponding to the to-be-processed frame, the previous frame of the to-be-processed frame, the reference frame, the previous frame of the reference frame, the audio data stream and the video data stream respectively, when it is determined that the blank PTS intervals corresponding to the audio data stream and the video data stream respectively are to be updated;

[0157] The second updating unit is used to update the PTS of the frame to be processed according to the blank PTS interval corresponding to the data stream to be processed.

[0158] In some embodiments, the first determining unit includes:

[0159] A first determining subunit, configured to determine a start time of a next unprocessed frame of the audio data stream based on a PTS of the next unprocessed frame of the audio data stream and a time unit adopted by the audio data stream;

[0160] A second determining subunit, configured to determine a start time of a next unprocessed frame of the video data stream based on a PTS of the next unprocessed frame of the video data stream and a time unit adopted by the video data stream;

[0161] The third determining subunit is used to determine, among the next unprocessed frame of the audio data stream and the next unprocessed frame of the video data stream, an unprocessed frame with a relatively small start time as a to-be-processed frame and an unprocessed frame with a relatively large start time as a reference frame.

[0162] In some embodiments, the second determining unit includes:

[0163] A first calculation subunit is used to calculate a first PTS interval and a second PTS interval, wherein the first PTS interval is: a PTS interval between a frame to be processed and a frame preceding the frame to be processed, and the second PTS interval is: a PTS interval between a reference frame and a frame preceding the reference frame;

[0164] a fourth determination subunit, configured to determine whether the first PTS interval is less than a preset first PTS interval threshold, and whether the second PTS interval is less than a preset second PTS interval threshold, wherein the first PTS interval threshold is determined based on a mean PTS interval corresponding to the data stream to be processed, and the second PTS interval threshold is determined based on a mean PTS interval corresponding to a reference data stream, wherein the reference data stream is: a data stream where the reference frame is located;

[0165] The fifth determining subunit determines to update the blank PTS intervals respectively corresponding to the audio data stream and the video data stream when the first PTS interval is not less than the first PTS interval threshold and the second PTS interval is not less than the second PTS interval threshold.

[0166] In some embodiments, the first updating unit includes:

[0167] a sixth determining subunit, configured to determine an end time of a previous frame of the frame to be processed, an end time of a previous frame of the reference frame, a start time of the frame to be processed, and a start time of the reference frame, wherein the end time of the previous frame of the frame to be processed and the end time of the previous frame of the reference frame are determined according to the mean values ​​of the PTS intervals corresponding to the audio data stream and the video data stream respectively;

[0168] A seventh determination subunit is used to use a relatively larger end time as a target end time between the end time of the previous frame of the frame to be processed and the end time of the previous frame of the reference frame;

[0169] an eighth determination subunit, configured to use a relatively smaller start time between the start time of the frame to be processed and the start time of the reference frame as a target start time;

[0170] A second calculation subunit is used to calculate the time interval of the blank interval according to the target end time and the target start time;

[0171] The updating subunit is used to update the blank PTS intervals corresponding to the audio and video data streams respectively according to the time interval of the blank interval.

[0172] In some embodiments, the audio and video processing device 10 further includes:

[0173] A third determination module is used to determine the time parameters of the audio and video data stream before the interruption when an interruption occurs during the process of receiving the audio and video data stream;

[0174] A fourth determination module is used to determine a target recovery video frame and a target recovery audio frame when the reception is restored, wherein the target recovery video frame is the first valid video frame received after the reception is restored, and the target recovery audio frame is the first valid audio frame received after the target recovery video frame;

[0175] An adjustment module is used to adjust the PTS of a target restored video frame and subsequent video frames according to a target restored video frame and time parameters, and to adjust the PTS of a target restored audio frame and subsequent audio frames according to a target restored audio frame and time parameters, wherein a subsequent video frame is a video frame received after a target restored video frame, and a subsequent audio frame is an audio frame received after a target restored audio frame.

[0176] In some embodiments, the third determination module includes:

[0177] A first determination submodule is used to determine a target interruption video frame and a target interruption audio frame, wherein the target interruption video frame is the last video frame carrying UTC time received before the interruption, and the target interruption audio frame is the last audio frame received before the interruption;

[0178] The second determination submodule is used to determine the time parameters according to the target interruption video frame and the target interruption audio frame, and the time parameters include: the PTS of the target interruption video frame, the PTS of the target interruption audio frame, the video frame reception completion time before the interruption and the audio frame reception completion time before the interruption, wherein the video frame reception completion time is determined according to the UTC time of the target interruption video frame, and the audio frame reception completion time is determined according to the time units respectively adopted by the audio and video data streams, the PTS of the target interruption audio frame, and the UTC time and PTS of the target interruption video frame.

[0179] In some embodiments, the fourth determining module includes:

[0180] A third determination submodule is used to determine whether the current video frame carries UTC time when the current video frame is received;

[0181] A first comparison submodule, for comparing the UTC time of the current video frame with the time when the video frame is received when the current video frame carries the UTC time;

[0182] A first discarding submodule is used to discard the current video frame if the current video frame does not carry UTC time, or the UTC time of the current video frame is not greater than the video frame reception completion time, and wait for receiving the next video frame until a target recovery video frame is determined;

[0183] The fourth determination submodule is used to determine the current video frame as a target restored video frame when the UTC time of the current video frame is greater than the video frame reception completion time.

[0184] In some embodiments, the fourth determining module further includes:

[0185] a fifth determination submodule, configured to determine the start time of the current audio frame if the current audio frame is received after the target restored video frame is determined;

[0186] A second comparison submodule, used to compare the start time of the current audio frame with the audio frame reception completion time;

[0187] A second discarding submodule is used to discard the current audio frame when the start time of the current audio frame is not greater than the audio frame reception completion time, and wait for receiving the next audio frame until the target restored audio frame is determined;

[0188] The sixth determination submodule is used to determine the current audio frame as a target restored audio frame when the start time of the current audio frame is greater than the audio frame reception completion time.

[0189] In some embodiments, the adjustment module includes:

[0190] A first calculation submodule, for calculating the video stream interruption PTS interval according to the PTS of the target restored video frame, the PTS of the target interrupted video frame and the average of the PTS intervals corresponding to the video data stream before the interruption;

[0191] The first adjustment submodule is used to adjust the PTS of the target restored video frame and subsequent video frames based on the video stream interruption PTS interval.

[0192] In some embodiments, the adjustment module further includes:

[0193] The second calculation submodule is used to calculate the audio stream interruption PTS interval according to the PTS of the target restored audio frame, the PTS of the target interrupted audio frame and the average of the PTS intervals corresponding to the audio data stream before the interruption;

[0194] The second adjustment submodule is used to adjust the PTS of the target restored audio frame and subsequent audio frames based on the audio stream interruption PTS interval.

[0195] As can be seen from the above, the embodiment of the present application compensates for the impact of missing data in the audio and video data stream on the playback by adjusting the PTS of the audio and video data stream. Among them, the mean PTS interval is the basis for the receiver to determine whether the audio and video data stream is continuous, which helps the receiver to find the blank interval; and the blank PTS interval is the basis for the receiver to process the PTS of the audio and video data stream, which helps to eliminate the impact of the blank interval on the audio and video data stream. Based on this, the receiver can use the mean PTS interval and the blank PTS interval as the basis for processing the PTS of the audio and video data stream frame by frame, and can update the blank PTS interval during the processing. After the PTS of the audio and video data stream is processed, since the impact of the blank interval on the audio and video data stream has been eliminated, the playback duration of each data frame determined according to the PTS of the audio and video data stream can keep the audio and video data stream continuous and synchronized during playback.

[0196] Corresponding to the audio and video processing method provided above, the present application embodiment also provides an electronic device. Fig.11 The electronic device 11 in the embodiment of the present application includes: a memory 1101, one or more processors 1102 ( Fig.11Only one is shown in the figure) and a computer program stored in the memory 1101 and executable on the processor. Among them: the memory 1101 is used to store software programs and units, and the processor 1102 executes various functional applications and data processing by running the software programs and units stored in the memory 1101 to obtain the resources corresponding to the above-mentioned preset events. Specifically, when the processor 1102 runs the above-mentioned computer program stored in the memory 1101, each step in the method embodiment is implemented, which will not be repeated here.

[0197] It should be understood that in the embodiment of the present application, the processor 1102 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0198] The memory 1101 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1102. A part or all of the memory 1101 may also include a non-volatile random access memory. For example, the memory 1101 may also store information on the device category.

[0199] As can be seen from the above, the embodiment of the present application compensates for the impact of missing data in the audio and video data stream on the playback by adjusting the PTS of the audio and video data stream. Among them, the mean PTS interval is the basis for the receiver to determine whether the audio and video data stream is continuous, which helps the receiver to find the blank interval; and the blank PTS interval is the basis for the receiver to process the PTS of the audio and video data stream, which helps to eliminate the impact of the blank interval on the audio and video data stream. Based on this, the receiver can use the mean PTS interval and the blank PTS interval as the basis for processing the PTS of the audio and video data stream frame by frame, and can update the blank PTS interval during the processing. After the PTS of the audio and video data stream is processed, since the impact of the blank interval on the audio and video data stream has been eliminated, the playback duration of each data frame determined according to the PTS of the audio and video data stream can keep the audio and video data stream continuous and synchronized during playback.

[0200] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0201] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0202] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0203] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0204] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0205] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the associated hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The above-mentioned computer-readable storage medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer-readable memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media does not include electrical carrier signals and telecommunication signals.

[0206] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An audio and video processing method, characterized in that: include: Determine the mean PTS intervals corresponding to the received audio and video data streams, respectively, wherein the audio and video data streams include an audio data stream and a video data stream, and the mean PTS interval is used to represent the mean of the inter-frame PTS interval values ​​of the corresponding data streams; Initialize the blank PTS intervals corresponding to the audio and video data streams respectively, wherein the blank PTS intervals are used to record the PTS intervals of the blank intervals of the corresponding data streams, and the blank intervals are intervals where no data frames exist in the audio and video data streams; According to the mean values ​​of the PTS intervals respectively corresponding to the audio and video data streams and the blank PTS intervals respectively corresponding to the audio and video data streams, the PTS of the audio and video data streams are processed frame by frame to eliminate the influence of the blank intervals on the audio and video data streams, wherein the blank PTS intervals respectively corresponding to the audio and video data streams are updated in the process of processing the PTS of the audio and video data streams; The playing duration of each data frame is determined according to the PTS of the audio and video data stream, so that the audio and video data stream remains continuous and synchronous during playing.

2. The audio and video processing method according to claim 1, characterized in that: The method of processing the PTS of the audio and video data stream frame by frame according to the mean values ​​of the PTS intervals respectively corresponding to the audio and video data streams and the blank PTS intervals respectively corresponding to the audio and video data streams comprises: In the case that both the audio data stream and the video data stream have unprocessed frames, a dual-channel data stream processing flow is triggered, wherein the dual-channel data stream processing flow is used to process the PTS of the audio data stream and the video data stream frame by frame according to the mean values ​​of the PTS intervals respectively corresponding to the audio data stream and the video data stream, and the blank PTS intervals respectively corresponding to the audio data stream and the video data stream; In the case that only one data stream among the audio data stream and the video data stream has the unprocessed frame, the single data stream processing flow is triggered, and the single data stream processing flow is used to process the PTS of the unprocessed data stream frame by frame according to the average PTS interval corresponding to the unprocessed data stream and the blank PTS interval corresponding to the unprocessed data stream, wherein the unprocessed data stream is: the data stream where the unprocessed frame is located.

3. The audio and video processing method according to claim 2, characterized in that: The dual-path data stream processing process includes: Determine a frame to be processed and a reference frame in a next unprocessed frame of the audio data stream and a next unprocessed frame of the video data stream; In the case where the frame to be processed is not the first frame of the data stream to be processed, determining whether to update the blank PTS intervals respectively corresponding to the audio data stream and the video data stream according to a preset update condition, the data stream to be processed being: the data stream where the frame to be processed is located; In the case where it is determined that the blank PTS intervals respectively corresponding to the audio data stream and the video data stream are to be updated, the blank PTS intervals respectively corresponding to the audio data stream and the video data stream are updated according to the mean values ​​of the PTS intervals respectively corresponding to the frame to be processed, the previous frame of the frame to be processed, the reference frame, the previous frame of the reference frame, the audio data stream and the video data stream; According to the blank PTS interval corresponding to the data stream to be processed, the PTS of the frame to be processed is updated.

4. The audio and video processing method according to claim 3, characterized in that: The step of determining a frame to be processed and a reference frame from the next unprocessed frame of the audio data stream and the next unprocessed frame of the video data stream comprises: Determine the start time of the next unprocessed frame of the audio data stream based on the PTS of the next unprocessed frame of the audio data stream and the time unit adopted by the audio data stream; Determine the start time of the next unprocessed frame of the video data stream based on the PTS of the next unprocessed frame of the video data stream and the time unit adopted by the video data stream; In the next unprocessed frame of the audio data stream and the next unprocessed frame of the video data stream, the unprocessed frame with a relatively small start time is determined as the to-be-processed frame, and the unprocessed frame with a relatively large start time is determined as the reference frame.

5. The audio and video processing method according to claim 3, characterized in that: Determining whether to update the blank PTS intervals respectively corresponding to the audio data stream and the video data stream according to the preset update condition includes: Calculating a first PTS interval and a second PTS interval, wherein the first PTS interval is: a PTS interval between the frame to be processed and a frame preceding the frame to be processed, and the second PTS interval is: a PTS interval between the reference frame and a frame preceding the reference frame; Determine whether the first PTS interval is less than a preset first PTS interval threshold, and whether the second PTS interval is less than a preset second PTS interval threshold, wherein the first PTS interval threshold is determined based on a mean PTS interval corresponding to the data stream to be processed, and the second PTS interval threshold is determined based on a mean PTS interval corresponding to a reference data stream, and the reference data stream is: a data stream where the reference frame is located; When the first PTS interval is not less than the first PTS interval threshold, and the second PTS interval is not less than the second PTS interval threshold, it is determined to update the blank PTS intervals respectively corresponding to the audio data stream and the video data stream.

6. The audio and video processing method according to claim 3, characterized in that: The updating of the blank PTS intervals corresponding to the audio data stream and the video data stream respectively according to the mean values ​​of the PTS intervals corresponding to the frame to be processed, the previous frame of the frame to be processed, the reference frame, the previous frame of the reference frame, the audio data stream and the video data stream respectively comprises: Determine the end time of the previous frame of the frame to be processed, the end time of the previous frame of the reference frame, the start time of the frame to be processed and the start time of the reference frame, wherein the end time of the previous frame of the frame to be processed and the end time of the previous frame of the reference frame are determined according to the average values ​​of the PTS intervals corresponding to the audio data stream and the video data stream respectively; Among the end time of the previous frame of the frame to be processed and the end time of the previous frame of the reference frame, the relatively larger end time is used as the target end time; Among the start time of the frame to be processed and the start time of the reference frame, the relatively smaller start time is used as the target start time; Calculating the time interval of the blank interval according to the target end time and the target start time; According to the time interval of the blank interval, the blank PTS intervals corresponding to the audio and video data streams are updated.

7. The audio and video processing method according to any one of claims 1 to 6, characterized in that: In the process of receiving the audio and video data stream, the audio and video processing method further includes: In the event of an interruption, determining a time parameter of the audio and video data stream before the interruption; In the case of recovery reception, determining a target recovery video frame and a target recovery audio frame, wherein the target recovery video frame is the first valid video frame received after the recovery reception, and the target recovery audio frame is the first valid audio frame received after the target recovery video frame; According to the target restored video frame and the time parameters, the PTS of the target restored video frame and subsequent video frames are adjusted, and according to the target restored audio frame and the time parameters, the PTS of the target restored audio frame and subsequent audio frames are adjusted, wherein the subsequent video frame is: a video frame received after the target restored video frame, and the subsequent audio frame is: an audio frame received after the target restored audio frame.

8. The audio and video processing method according to claim 7, characterized in that: The determining of the time parameter of the audio and video data stream before the interruption includes: Determine a target interruption video frame and a target interruption audio frame, wherein the target interruption video frame is the last video frame carrying UTC time received before the interruption, and the target interruption audio frame is the last audio frame received before the interruption; According to the target interruption video frame and the target interruption audio frame, time parameters are determined, and the time parameters include: the PTS of the target interruption video frame, the PTS of the target interruption audio frame, the video frame reception completion time before the interruption and the audio frame reception completion time before the interruption, wherein the video frame reception completion time is determined according to the UTC time of the target interruption video frame, and the audio frame reception completion time is determined according to the time units respectively adopted by the audio and video data streams, the PTS of the target interruption audio frame, and the UTC time and PTS of the target interruption video frame.

9. The audio and video processing method according to claim 8, characterized in that: The determining of the target restored video frame and the target restored audio frame comprises: When receiving a current video frame, determining whether the current video frame carries UTC time; In the case where the current video frame carries UTC time, comparing the UTC time of the current video frame with the time when the video frame is received; If the current video frame does not carry the UTC time, or the UTC time of the current video frame is not greater than the video frame reception completion time, discard the current video frame, and wait for receiving the next video frame until the target restored video frame is determined; In a case where the UTC time of the current video frame is greater than the video frame reception completion time, the current video frame is determined as the target recovery video frame.

10. The audio and video processing method according to claim 8, characterized in that: The determining of the target restored video frame and the target restored audio frame comprises: After determining the target restored video frame, if a current audio frame is received, determining a start time of the current audio frame; Compare the start time of the current audio frame with the completion time of receiving the audio frame; If the start time of the current audio frame is not greater than the audio frame reception completion time, discard the current audio frame and wait for receiving the next audio frame until the target restored audio frame is determined; In a case where the start time of the current audio frame is greater than the audio frame reception completion time, the current audio frame is determined as the target restored audio frame.

11. The audio and video processing method according to claim 8, characterized in that: The time parameter also includes: the mean value of the PTS interval corresponding to the video data stream before the interruption; and adjusting the PTS of the target restored video frame and subsequent video frames according to the target restored video frame and the time parameter, including: Calculate the video stream interruption PTS interval according to the PTS of the target restored video frame, the PTS of the target interrupted video frame and the average of the PTS intervals corresponding to the video data stream before the interruption; Based on the video stream interruption PTS interval, the PTS of the target recovery video frame and the subsequent video frame are adjusted.

12. The audio and video processing method according to claim 8, characterized in that: The time parameter also includes: the average PTS interval corresponding to the audio data stream before the interruption; the PTS of the target restored audio frame and subsequent audio frames are adjusted according to the target restored audio frame and the time parameter, including: Calculate the audio stream interruption PTS interval according to the PTS of the target restored audio frame, the PTS of the target interrupted audio frame and the average of the PTS intervals corresponding to the audio data stream before the interruption; Based on the audio stream interruption PTS interval, the PTS of the target recovery audio frame and the subsequent audio frame are adjusted.

13. An audio and video processing device, characterized in that: include: A first determination module is used to determine the mean PTS intervals corresponding to the received audio and video data streams, wherein the audio and video data streams include audio data streams and video data streams, and the mean PTS intervals are used to represent the mean of the inter-frame PTS intervals of the corresponding data streams; An initialization module, used to initialize the blank PTS intervals corresponding to the audio and video data streams, respectively, wherein the blank PTS intervals are used to record the PTS intervals of the blank intervals of the corresponding data streams, and the blank intervals are intervals where no data frames exist in the audio and video data streams; A processing module, for processing the PTS of the audio and video data stream frame by frame according to the mean values ​​of the PTS intervals respectively corresponding to the audio and video data streams and the blank PTS intervals respectively corresponding to the audio and video data streams, so as to eliminate the influence of the blank intervals on the audio and video data streams, wherein the blank PTS intervals respectively corresponding to the audio and video data streams are updated in the process of processing the PTS of the audio and video data streams; The second determination module is used to determine the playing time of each data frame according to the PTS of the audio and video data stream, so that the audio and video data stream remains continuous and synchronous during playing.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 12 is implemented.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 12 is implemented.

Images