Timestamp optimization method and device in video transcoding process, computer equipment and readable storage medium

Abstract

The invention relates to a timestamp optimization method and device in a video transcoding process, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: reading video code stream coding data of a source video; the video code stream coding data comprises a current video frame; filtering the display timestamp associated with the current video frame to obtain a filtered display timestamp corresponding to the current video frame; storing the filtered display timestamps into a buffer queue according to a time sequence; the buffer queue stores the filtered display timestamps sorted according to a monotone increasing sequence; and after the video encoder outputs the transcoded current video frame, determining a decoding timestamp corresponding to the transcoded current video frame in the buffer queue. By adopting the method, the stability of the video frame spacing can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, device, computer equipment, computer-readable storage medium and computer program product for optimizing timestamps in a video transcoding process. Background Art

[0002] With the continuous development of communication technology and the Internet, it has become a normal practice for users to play videos or watch live broadcasts through mobile devices. Video transcoding is commonly used in live broadcast and on-demand services, and is mainly used to process source videos to meet the needs of playback on different forms of network terminals.

[0003] However, due to various reasons, the source stream may be disconnected and re-pushed, the frame rate may change dynamically, and video frames may be lost due to network reasons. These abnormal problems will challenge the processing of timestamps. If the timestamps are converted according to the initial fixed timestamps, the timestamps after transcoding will be unevenly distributed, resulting in irregular changes in the spacing between video frames after transcoding, affecting the robustness of the video on the user's playback end.

[0004] Therefore, there is a problem of uneven distribution of timestamps after transcoding in the related art. Summary of the invention

[0005] Based on this, it is necessary to provide a timestamp optimization method, device, computer equipment, computer-readable storage medium and computer program product in the video transcoding process that can improve the smoothness of video frame spacing in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for optimizing timestamps during video transcoding, comprising:

[0007] Read the video code stream encoding data of the source video; the video code stream encoding data includes the current video frame;

[0008] Performing filtering processing on the display timestamp associated with the current video frame to obtain a filtered display timestamp corresponding to the current video frame;

[0009] storing the filtered display timestamps in a buffer queue in chronological order; the buffer queue stores the filtered display timestamps sorted in a monotonically increasing order;

[0010] After the video encoder outputs the transcoded current video frame, a decoding timestamp corresponding to the transcoded current video frame is determined in the buffer queue.

[0011] In one of the embodiments, determining in the buffer queue a decoding timestamp corresponding to the transcoded current video frame includes:

[0012] Taking out the first filtered display timestamp from the buffer queue;

[0013] The first filtered display timestamp is used as the decoding timestamp corresponding to the transcoded current video frame.

[0014] In one embodiment, the number of display timestamps associated with the current video frame is multiple, and filtering the display timestamp associated with the current video frame to obtain the filtered display timestamp corresponding to the current video frame includes:

[0015] According to a preset normalization kernel, a plurality of display timestamps associated with the current video frame are filtered to obtain a filtered display timestamp corresponding to the current video frame.

[0016] In one embodiment, the method further comprises:

[0017] Obtaining a current display timestamp corresponding to the current video frame, and caching the current display timestamp into a display timestamp list corresponding to the source video;

[0018] According to the radius of the preset normalized kernel, a display timestamp sub-list associated with the current display timestamp is screened out from the display timestamp list; the display timestamp sub-list is centered on the current display timestamp and includes display timestamps before and after the current display timestamp; the amount of data in the display timestamp sub-list matches the radius;

[0019] The display timestamp in the display timestamp sublist associated with the current display timestamp is used as the display timestamp associated with the current video frame.

[0020] In one of the embodiments, when the amount of data in the display timestamp list does not match the radius of the normalization kernel, filtering out a display timestamp sub-list associated with the current display timestamp from the display timestamp list according to the preset radius of the normalization kernel includes:

[0021] Within the radius of the normalized kernel, selecting a display timestamp associated with the current display timestamp from a display timestamp list;

[0022] Acquire a real-time frame spacing that matches the current video frame, and determine a predicted display timestamp associated with the current display timestamp according to the real-time frame spacing and the filtered display timestamp;

[0023] The display timestamp sub-list is composed according to the display timestamp associated with the current display timestamp and the predicted display timestamp.

[0024] In one embodiment, the video code stream encoding data also includes a previous video frame, and obtaining a real-time frame spacing matching the current video frame includes:

[0025] Obtaining a current frame spacing according to a current decoding timestamp corresponding to the current video frame and a previous decoding timestamp corresponding to the previous video frame;

[0026] The real-time frame spacing is estimated according to the current frame spacing.

[0027] In a second aspect, the present application further provides a timestamp optimization device during video transcoding, comprising:

[0028] A reading module, used for reading the video code stream encoding data of the source video; the video code stream encoding data includes the current video frame;

[0029] A filtering module, configured to filter the display timestamp associated with the current video frame to obtain a filtered display timestamp corresponding to the current video frame;

[0030] A storage module, used for storing the filtered display timestamps into a buffer queue in chronological order; the buffer queue stores the filtered display timestamps sorted in a monotonically increasing order;

[0031] The fetching module is used to determine the decoding timestamp corresponding to the transcoded current video frame in the buffer queue after the video encoder outputs the transcoded current video frame.

[0032] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0034] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0035] The timestamp optimization method, device, computer equipment, computer-readable storage medium and computer program product in the above-mentioned video transcoding process read the video code stream encoding data of the source video; the video code stream encoding data includes the current video frame, and the display timestamp associated with the current video frame is filtered to obtain the filtered display timestamp corresponding to the current video frame; the filtered display timestamp is stored in a buffer queue in chronological order; the buffer queue stores the filtered display timestamps sorted in a monotonically increasing order; after the video encoder outputs the transcoded current video frame, the decoding timestamp corresponding to the transcoded current video frame is determined in the buffer queue.

[0036] In this way, the display timestamp associated with the current video frame is smoothed by filtering, and the filtered display timestamp is stored in the buffer queue in chronological order. After the video encoder outputs the transcoded current video frame, the decoding timestamp corresponding to the transcoded current video frame is determined in the buffer queue storing the filtered display timestamps sorted in monotonically increasing order, so that the spacing between transcoded video frames is maintained within a relatively stable range. Especially in the scenario where the source video frame rate fluctuates, the spacing between the timestamps of the transcoded frames can be well smoothed, and abnormal problems such as stuttering at the playback end caused by timestamp problems can be solved, thereby effectively reducing the problem of irregular changes in the spacing between transcoded video frames caused by irregular timestamp fluctuations before transcoding, thereby improving the robustness of the video at the user playback end. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A schematic diagram of a flow chart of a timestamp optimization method in a video transcoding process in one embodiment;

[0039] Figure 2 A schematic diagram of a DTS timestamp buffering process in one embodiment;

[0040] Figure 3 A schematic diagram of a flow chart of yet another method for optimizing timestamps in a video transcoding process in an embodiment;

[0041] Figure 4 A schematic flow chart of a method for optimizing timestamps in a video transcoding process in another embodiment;

[0042] Figure 5It is a structural block diagram of a timestamp optimization device in a video transcoding process in one embodiment;

[0043] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0046] In one embodiment, Figure 1 As shown, a method for optimizing timestamps during video transcoding is provided. This embodiment uses the method applied to a computer device as an example. It can be understood that the computer device can be a terminal, a server, or a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and Internet of Things devices. The server can be implemented as an independent server or a server cluster composed of multiple servers. In this embodiment, the method includes the following steps:

[0047] Step S110, reading the video code stream encoding data of the source video.

[0048] The video code stream encoding data may refer to an encoded video data sequence.

[0049] The video code stream encoding data includes the current video frame.

[0050] In a specific implementation, the computer device can read the video code stream encoding data of the source video.

[0051] Step S120 , filtering the display timestamp associated with the current video frame to obtain a filtered display timestamp corresponding to the current video frame.

[0052] The display timestamp is PTS (Presentation Time Stamp), which refers to the time when the video frame should be displayed.

[0053] In a specific implementation, the computer device may decode the current video frame to further obtain a display timestamp associated with the current video frame, thereby filtering the display timestamp associated with the current video frame to obtain a filtered display timestamp corresponding to the current video frame.

[0054] In some embodiments, there may be multiple display timestamps associated with the current video frame, including not only the current display timestamp corresponding to the current video frame obtained by decoding the current video frame, but also display timestamps before and after the current display timestamp. By filtering the multiple display timestamps associated with the current video frame, the filtered display timestamp corresponding to the current video frame is obtained.

[0055] After the current video frame is re-encoded, the filtered display timestamp corresponding to the current video frame can be used as the display timestamp corresponding to the re-encoded current video frame.

[0056] Step S130, storing the filtered display timestamps into a buffer queue in chronological order.

[0057] The buffer queue stores the filtered display timestamps sorted in a monotonically increasing order.

[0058] In a specific implementation, the computer device may store the filtered display timestamps into a buffer queue in chronological order.

[0059] Step S140: after the video encoder outputs the transcoded current video frame, a decoding timestamp corresponding to the transcoded current video frame is determined in the buffer queue.

[0060] The transcoded current video frame may be a current video frame that is re-encoded and packaged into a target format.

[0061] In a specific implementation, after the video encoder outputs the transcoded current video frame, the computer device may determine the decoding timestamp corresponding to the transcoded current video frame in the buffer queue according to the time sequence of the filtered display timestamps.

[0062] In the above-mentioned timestamp optimization method during video transcoding, the video code stream encoding data of the source video is read; the video code stream encoding data includes the current video frame, and the display timestamp associated with the current video frame is filtered to obtain the filtered display timestamp corresponding to the current video frame; the filtered display timestamp is stored in a buffer queue in chronological order; the buffer queue stores the filtered display timestamps sorted in a monotonically increasing order; after the video encoder outputs the transcoded current video frame, the decoding timestamp corresponding to the transcoded current video frame is determined in the buffer queue.

[0063] In this way, the display timestamp associated with the current video frame is smoothed by filtering, and the filtered display timestamp is stored in the buffer queue in chronological order. After the video encoder outputs the transcoded current video frame, the decoding timestamp corresponding to the transcoded current video frame is determined in the buffer queue storing the filtered display timestamps sorted in monotonically increasing order, so that the spacing between transcoded video frames is maintained within a relatively stable range. Especially in the scenario where the source video frame rate fluctuates, the spacing between the timestamps of the transcoded frames can be well smoothed, and abnormal problems such as stuttering at the playback end caused by timestamp problems can be solved, thereby effectively reducing the problem of irregular changes in the spacing between transcoded video frames caused by irregular timestamp fluctuations before transcoding, thereby improving the robustness of the video at the user playback end.

[0064] In one embodiment, determining the decoding timestamp corresponding to the current transcoded video frame in the buffer queue includes: taking out the first filtered display timestamp in the buffer queue; and using the first filtered display timestamp as the decoding timestamp corresponding to the current transcoded video frame.

[0065] Among them, the decoding timestamp (DTS) is used to tell the decoder when to decode a frame of compressed video data.

[0066] In a specific implementation, when the computer device determines the decoding timestamp corresponding to the current transcoded video frame in the buffer queue, the computer device can take out the first filtered display timestamp in the buffer queue; and use the first filtered display timestamp as the decoding timestamp corresponding to the current transcoded video frame.

[0067] In this way, every time the video encoder outputs a transcoded video frame, the first (smallest) value can be taken out from the buffer queue as its decoding timestamp, thereby obtaining a monotonic and uniform decoding timestamp, ensuring that the decoding timestamps corresponding to the output transcoded video frames are assigned in ascending order, thereby ensuring the equidistant monotonically increasing property of the decoding timestamps.

[0068] In order to facilitate the understanding of those skilled in the art, Figure 2The DTS timestamp buffering process when the number of B frames (Bi-directional Predicted Frame, bidirectional predictive interpolation coded frame, which compresses data by referring to the previous frame and the next frame that have been coded at the same time. B frame records the residual information with the previous and next frames, and the data of the previous and next frames need to be referred to when decoding) is 2 is given. The video frame display sequence 0~n represents the filtered display timestamps corresponding to n video frames (I frame, B frame, B frame, P frame, B frame, B frame, P frame, etc.) of the video code stream coded data. Among them, I frame (Intra-coded Frame) is a key frame, a type of conventional frame, which contains complete information of the video frame and can be decoded independently; P frame (Predictive Frame) is a forward predictive coded frame, which compresses the current frame by referring to the previously coded frame, and records the residual information with the reference frame. The brown table represents the filtered display timestamp queue (i.e., buffer queue) sorted by time. The data with PTS of 0 is initially put into the buffer queue, and the video encoder has no output frame at this time. Since frames 1 and 2 are B frames, the data with PTS 3 corresponding to the P frame enters the buffer queue, and the buffer queue is rearranged in ascending order. At this time, the minimum value is 0. If the video encoder has transcoded video frames output, take 0 out of the queue; the data with PTS 1 enters the buffer queue. After reordering, the data in the buffer queue is 1 and 3. If there is transcoded video frames output, take the minimum value 1 out of the queue, and so on. This ensures that the DTS corresponding to the transcoded video frames output by the video encoder are assigned in ascending order.

[0069] The technical solution of this embodiment is to take out the first filtered display timestamp from the buffer queue and use the first filtered display timestamp as the decoding timestamp corresponding to the transcoded current video frame. In this way, the first filtered display timestamp after reordering the filtered display timestamps is used as the DTS of the transcoded current video frame, which ensures the equidistant monotonically increasing property of the DTS timestamp corresponding to the output transcoded video frame, so that the frame spacing is maintained within a relatively stable range.

[0070] In one embodiment, there are multiple display timestamps associated with the current video frame, and the display timestamps associated with the current video frame are filtered to obtain a filtered display timestamp corresponding to the current video frame, including: filtering multiple display timestamps associated with the current video frame according to a preset normalization kernel to obtain a filtered display timestamp corresponding to the current video frame.

[0071] In a specific implementation, the number of display timestamps associated with the current video frame is multiple, including not only the current display timestamp corresponding to the current video frame obtained by decoding the current video frame, but also the display timestamps before and after the current display timestamp. The multiple display timestamps associated with the current video frame are filtered by a preset normalization kernel to obtain the filtered display timestamp corresponding to the current video frame.

[0072] In the technical solution of this embodiment, the number of display timestamps associated with the current video frame is multiple, and the multiple display timestamps associated with the current video frame are filtered according to a preset normalization kernel to obtain the filtered display timestamp corresponding to the current video frame. In this way, by filtering the multiple display timestamps associated with the current video frame through a preset normalization kernel, the multiple display timestamps can be weighted, thereby significantly reducing the jitter of the timestamps, more effectively ensuring that the timestamps of the transcoded video frames increase steadily, making the video playback smoother, and avoiding the freeze or frame skipping phenomenon caused by the timestamp fluctuation.

[0073] In one embodiment, the method further includes: obtaining a current display timestamp corresponding to the current video frame, and caching the current display timestamp in a display timestamp list corresponding to the source video; filtering out a display timestamp sublist associated with the current display timestamp in the display timestamp list according to a preset normalized kernel radius; and using the display timestamp in the display timestamp sublist associated with the current display timestamp as the display timestamp associated with the current video frame.

[0074] The display timestamp sublist is centered on the current display timestamp and includes the display timestamps before and after the current display timestamp.

[0075] The number of data in the display timestamp sublist matches the radius.

[0076] The display timestamp list corresponding to the source video is used to cache the display timestamps corresponding to each video frame of the source video.

[0077] In a specific implementation, before the step of filtering the multiple display timestamps associated with the current video frame according to a preset normalization kernel, the computer device can obtain the current display timestamp corresponding to the current video frame, cache the current display timestamp in a display timestamp list corresponding to the source video, and filter out a display timestamp sublist associated with the current display timestamp in the display timestamp list according to the radius of the preset normalization kernel; the display timestamp sublist is centered on the current display timestamp and includes display timestamps before and after the current display timestamp; wherein the amount of data in the display timestamp sublist matches the radius.

[0078] Among them, the number of data in the display timestamp sublist matches the radius, specifically, the number of display timestamps in the display timestamp sublist that are located before the current display timestamp is equal to the number of display timestamps in the display timestamp sublist that are located after the current display timestamp, and both are equal to the radius of the preset normalized kernel.

[0079] In this way, the computer device may use the display timestamp in the display timestamp sub-list associated with the current display timestamp as the display timestamp associated with the current video frame.

[0080] Therefore, when the computer device performs filtering processing on multiple display time stamps associated with the current video frame according to the preset normalization kernel, the computer device may use the following formula:

[0081] = ;

[0082] in, represents the normalized kernel, A list of display timestamps representing the cached source videos. Represents the filtered display timestamp sublist, Indicates the current index of the timestamp list. represents the radius of the normalized kernel.

[0083] For example, to display the current index of a timestamp list Taking the index corresponding to the current display timestamp as an example, assuming that the display timestamp list of the source video is The cache has 11 display timestamps [x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10]. =5, which means that the current display timestamp is the display timestamp list For the fifth display timestamp x4 in the image, if the radius of the normalized kernel is 3, the first three display timestamps (x1, x2, x3) and the last three display timestamps (x5, x6, x7) of x4 are selected. Therefore, the display timestamp sublist associated with the current display timestamp x4 is [x1, x2, x3, x4, x5, x6, x7].

[0084] There may be multiple candidate normalized kernels, and one normalized kernel may be specified from the multiple candidate normalized kernels. For example, a normalized kernel with the longest radius may be specified as a preset normalized kernel.

[0085] For example, the candidate normalized kernels with radii of 1, 2, and 3 can be as follows:

[0086] = ;

[0087] = ;

[0088] = ;

[0089] In some embodiments, the longest radius may be selected. As the default normalization kernel.

[0090] In other embodiments, each candidate normalization kernel may be used to filter the display timestamps in the display timestamp list corresponding to the source video, and the normalization kernel with the best filtering effect may be selected as the preset normalization kernel.

[0091] The technical solution of this embodiment obtains the current display timestamp corresponding to the current video frame, and caches the current display timestamp in the display timestamp list corresponding to the source video; according to the radius of the preset normalization kernel, a display timestamp sublist associated with the current display timestamp is screened out in the display timestamp list; the display timestamp sublist is centered on the current display timestamp and includes display timestamps before and after the current display timestamp; the number of data in the display timestamp sublist matches the radius; and the display timestamp in the display timestamp sublist associated with the current display timestamp is used as the display timestamp associated with the current video frame.

[0092] In this way, through the preset radius of the normalization kernel, the display timestamps that are close to the time before and after the current display timestamp are selected from the display timestamp list corresponding to the source video to form a display timestamp sublist associated with the current display timestamp. The number of data in the display timestamp sublist matches the radius of the normalization kernel. Therefore, in the process of filtering the display timestamp sublist through the preset normalization kernel, the error caused by the fluctuation of a single timestamp can be reduced, making the video playback smoother and reducing the freeze or frame skipping caused by unstable timestamps. Especially in fast-moving or changing scenes, using surrounding timestamps can better capture the trend of time changes and provide more accurate time information.

[0093] In one embodiment, when the amount of data in the display timestamp list does not match the radius of the normalization kernel, a display timestamp sub-list associated with the current display timestamp is filtered out in the display timestamp list according to the preset radius of the normalization kernel, including: within the radius of the normalization kernel, filtering out the display timestamp associated with the current display timestamp in the display timestamp list; obtaining a real-time frame spacing that matches the current video frame, and determining a predicted display timestamp associated with the current display timestamp according to the real-time frame spacing and the filtered display timestamp; and composing a display timestamp sub-list according to the display timestamp associated with the current display timestamp and the predicted display timestamp.

[0094] For example, a list of source video display timestamps The cache has 5 display timestamps [x0, x1, x2, x3, x4], so the number of data in the display timestamp list is 5; the display timestamp list of the source video The cache has 6 display timestamps [x0, x1, x2, x3, x4, x5], so the number of data in the display timestamp list is 6.

[0095] The predicted display timestamp is a display timestamp predicted according to the real-time frame spacing.

[0096] Among them, when the number of data in the display timestamp list does not match the radius of the normalization kernel, the computer device, in the process of filtering out the display timestamp sub-list associated with the current display timestamp in the display timestamp list according to the preset radius of the normalization kernel, can filter out the display timestamp associated with the current display timestamp in the display timestamp list within the radius of the normalization kernel; then obtain the real-time frame spacing matching the current video frame, and determine the predicted display timestamp associated with the current display timestamp according to the real-time frame spacing and the filtered display timestamp; in this way, a display timestamp sub-list can be formed according to the display timestamp associated with the current display timestamp and the predicted display timestamp.

[0097] For example, a list of source video display timestamps The cache has 5 display timestamps [x0, x1, x2, x3, x4]. If the radius of the normalization kernel is 3, the display timestamp sublist associated with the current display timestamp needs to have 7 display timestamps. Therefore, the number of data in the display timestamp list is less than the number of display timestamp lists required by the radius of the normalization kernel, that is, the number of data in the display timestamp list does not match the radius of the normalization kernel. The computer device can supplement the display timestamps according to the real-time frame spacing.

[0098] For example, =3, which means displaying the timestamp list For the third display timestamp x2 in the normalized kernel, if the radius of the normalized kernel is 3, first filter out the display timestamps associated with the current display timestamp x2 in the display timestamp list within the radius of the normalized kernel: x0, x1, x2, x3, x4.

[0099] Then, according to the real-time frame spacing and the screened display timestamps, determine the predicted display timestamp associated with the current display timestamp. Specifically, there are 5 display timestamps associated with the current display timestamp screened out in the display timestamp list, and the display timestamp sub-list associated with the current display timestamp needs to have 7 display timestamps, so it is necessary to add a predicted display timestamp before and after the current display timestamp. Assuming that the real-time frame spacing matched by the current video frame is estimate_duration, the predicted display timestamp before the current display timestamp is: x0-estimate_duration; the predicted display timestamp after the current display timestamp is: x4+ estimate_duration. In this way, according to the display timestamp associated with the current display timestamp and the predicted display timestamp, a display timestamp sub-list is formed: [x0-estimate_duration, x0, x1, x2, x3, x4, x4+ estimate_duration].

[0100] For example, the display timestamp list of the source video The cache has 10 display timestamps [x0, x1, x2, x3, x4, x5, x6, x8, x9]. If the radius of the normalized kernel is 3, =7, which means displaying the timestamp list For the 7th display timestamp x6, it is necessary to filter out the first 3 display timestamps of x6 and the last 3 display timestamps of x6 in the display timestamp list of the source video. However, x10 does not exist in the display timestamp list of the source video. Therefore, the number of data in the display timestamp list does not match the radius of the normalized kernel, and the display timestamps need to be supplemented according to the real-time frame spacing.

[0101] First, within the radius of the normalized kernel, filter out the display timestamps associated with the current display timestamp x6 in the display timestamp list: x3, x4, x5, x6, x8, x9.

[0102] Then, based on the real-time frame spacing and the screened display timestamps, determine the predicted display timestamp associated with the current display timestamp. Specifically, 6 display timestamps associated with the current display timestamp are screened out in the display timestamp list, and the display timestamp sub-list associated with the current display timestamp needs to have 7 display timestamps, and a predicted display timestamp x10 needs to be added after the current display timestamp. Assuming that the real-time frame spacing matched by the current video frame is estimate_duration, the predicted display timestamp x10 located after the current display timestamp is: x9+estimate_duration. In this way, based on the display timestamps associated with the current display timestamp and the predicted display timestamps, a display timestamp sub-list is formed: [x3, x4, x5, x6, x8, x9, x9+ estimate_duration].

[0103] The technical solution of this embodiment is as follows: when the number of data in the display timestamp list does not match the radius of the normalization kernel, the display timestamp associated with the current display timestamp is filtered out in the display timestamp list within the radius of the normalization kernel; a real-time frame spacing that matches the current video frame is obtained, and a predicted display timestamp associated with the current display timestamp is determined based on the real-time frame spacing and the filtered display timestamp; and a display timestamp sub-list is composed based on the display timestamp associated with the current display timestamp and the predicted display timestamp.

[0104] In this way, when the data quantity of the display timestamp list does not match the radius of the normalization kernel, the display timestamp is supplemented by the real-time frame spacing that matches the current video frame to obtain the predicted display timestamp associated with the current display timestamp, so that the data quantity of the display timestamp sublist composed of the display timestamp associated with the current display timestamp and the predicted display timestamp can match the radius of the normalization kernel, so that the display timestamp lists of different lengths can be flexibly processed, ensuring that even in the case of insufficient data, a valid display timestamp sublist associated with the current display timestamp can still be generated, thereby improving the reliability of the display timestamp sublist. At the same time, the real-time frame spacing corresponding to the current video frame is obtained and combined with the display timestamps selected in the display timestamp list within the radius of the normalization kernel, so that the display timestamp associated with the current display timestamp can be more accurately predicted, thereby improving the reliability of the predicted display timestamp.

[0105] In one embodiment, the video code stream encoding data also includes a previous video frame, and obtaining a real-time frame spacing that matches the current video frame includes: obtaining the current frame spacing based on a current decoding timestamp corresponding to the current video frame and a previous decoding timestamp corresponding to the previous video frame; and estimating the real-time frame spacing based on the current frame spacing.

[0106] In a specific implementation, the video code stream encoding data also includes the previous video frame. When the computer device is in the process of obtaining the real-time frame spacing that matches the current video frame, the computer device can obtain the current frame spacing based on the current decoding timestamp corresponding to the current video frame and the previous decoding timestamp corresponding to the previous video frame; thereby estimating the real-time frame spacing corresponding to the current video frame based on the current frame spacing.

[0107] The technical solution of this embodiment obtains the current frame spacing according to the current decoding timestamp corresponding to the current video frame and the previous decoding timestamp corresponding to the previous video frame; and estimates the real-time frame spacing according to the current frame spacing. In this way, the current frame spacing is obtained by using the decoding timestamps of the current frame and the previous frame. The real-time frame spacing estimated according to the current frame spacing can more accurately reflect the actual frame spacing in the video stream, reduce errors caused by fluctuations or abnormal values, and can flexibly adapt to the dynamic changes of the video content and adjust the frame spacing estimation in real time. This real-time frame spacing estimation method can better reflect the state of the current video stream.

[0108] In order to facilitate the understanding of those skilled in the art, Figure 3 Another flowchart of timestamp optimization during video transcoding is provided. Figure 3 As shown, 1) the computer device decapsulates the source video, reads the corresponding video code stream encoding data (video frame data) to be processed, and parses the DTS timestamp.

[0109] 2) The computer device estimates the real-time frame spacing (and frame rate) of the source video in real time based on the DTS timestamp. Specifically, the current frame spacing is obtained based on the current decoding timestamp corresponding to the current video frame and the previous decoding timestamp corresponding to the previous video frame; the frame spacing is estimated based on the current frame spacing.

[0110] Among them, the current frame interval duration = current_dts - last_dts.

[0111] Among them, current_dts represents the current decoding timestamp, and last_dts represents the last decoding timestamp.

[0112] The estimated real-time frame interval estimate_duration = estimate_duration * (1-α) + duration * α, where α is a floating point number much smaller than 1.

[0113] Estimated current frame rate estimate_framerate = 1 / estimate_duration.

[0114] 3) Decode the current video frame and obtain the decoded image and the corresponding PTS timestamp.

[0115] 4) Filter the PTS timestamp:

[0116] = ;

[0117] 5) Encode the current video frame, filtered The timestamp is used as the corresponding time of the re-encoded current video frame. Timestamp.

[0118] 6) After filtering The timestamps are put into a buffer queue and maintained in a monotonically increasing order within the buffer queue.

[0119] 7) When the video encoder outputs the transcoded current video frame, the first (smallest) value is taken out from the buffer queue described in step 6) as its DTS timestamp.

[0120] 8) The transcoded video frames output by the video encoder are written to the output stream.

[0121] As shown in Table 1 below, the PTS timestamps of the first 8 frames of a video are captured. N1~N8 represent the PTS timestamps corresponding to the video frames. The second row of the table represents the timestamps corresponding to the video frames in the source stream. K1~K3 represent the timestamps of each frame after filtering using different normalized kernels. It can be seen that after filtering, the output frame interval is smoother than the source timestamp.

[0122] Table 1 PTS timestamps of the first 8 frames of a video

[0123]

[0124] In another embodiment, Figure 4 As shown, a method for optimizing timestamps in a video transcoding process is provided, and the method is described by taking the application of the method to a computer device as an example, and the method includes the following steps:

[0125] Step S402, reading the video code stream encoding data of the source video.

[0126] Step S404, obtaining a current display timestamp corresponding to the current video frame, and caching the current display timestamp into a display timestamp list corresponding to the source video.

[0127] Step S406: According to the preset normalized kernel radius, a display timestamp sub-list associated with the current display timestamp is filtered out from the display timestamp list.

[0128] Step S408: Use the display timestamp in the display timestamp sublist associated with the current display timestamp as the display timestamp associated with the current video frame.

[0129] Step S410 , filtering is performed on multiple display timestamps associated with the current video frame according to a preset normalization kernel to obtain a filtered display timestamp corresponding to the current video frame.

[0130] Step S412: storing the filtered display timestamps into a buffer queue in chronological order.

[0131] Step S414: after the video encoder outputs the transcoded current video frame, the first filtered display timestamp is retrieved from the buffer queue.

[0132] Step S416: Use the first filtered display timestamp as the decoding timestamp corresponding to the transcoded current video frame.

[0133] It should be noted that the specific limitations of the above steps can refer to the specific limitations of a timestamp optimization method in a video transcoding process described above.

[0134] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0135] Based on the same inventive concept, the embodiment of the present application also provides a device for optimizing timestamps in a video transcoding process for implementing the above-mentioned method for optimizing timestamps in a video transcoding process. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more devices for optimizing timestamps in a video transcoding process provided below can refer to the limitations of the method for optimizing timestamps in a video transcoding process above, and will not be repeated here.

[0136] In an exemplary embodiment, Figure 5 As shown, a timestamp optimization device in a video transcoding process is provided, comprising: a reading module 510, a filtering module 520, a storage module 530 and a determination module 540, wherein:

[0137] The reading module 510 is used to read the video code stream encoding data of the source video; the video code stream encoding data includes the current video frame.

[0138] The filtering module 520 is configured to perform filtering processing on the display timestamp associated with the current video frame to obtain a filtered display timestamp corresponding to the current video frame.

[0139] The storage module 530 is used to store the filtered display timestamps into a buffer queue in time sequence; the buffer queue stores the filtered display timestamps sorted in a monotonically increasing order.

[0140] The determination module 540 is configured to determine, after the video encoder outputs the transcoded current video frame, a decoding timestamp corresponding to the transcoded current video frame in the buffer queue.

[0141] In one of the embodiments, the determination module 540 is specifically configured to retrieve a first filtered display timestamp from the buffer queue; and use the first filtered display timestamp as a decoding timestamp corresponding to the transcoded current video frame.

[0142] In one embodiment, the number of display timestamps associated with the current video frame is multiple, and the filtering module 520 is specifically used to filter the multiple display timestamps associated with the current video frame according to a preset normalization kernel to obtain a filtered display timestamp corresponding to the current video frame.

[0143] In one of the embodiments, the device also includes: an association determination module, used to obtain a current display timestamp corresponding to the current video frame, and cache the current display timestamp in a display timestamp list corresponding to the source video; according to the radius of the preset normalization kernel, filter out a display timestamp sublist associated with the current display timestamp in the display timestamp list; the display timestamp sublist is centered on the current display timestamp and includes display timestamps before and after the current display timestamp; the amount of data in the display timestamp sublist matches the radius; and the display timestamp in the display timestamp sublist associated with the current display timestamp is used as the display timestamp associated with the current video frame.

[0144] In one of the embodiments, when the amount of data in the display timestamp list does not match the radius of the normalization kernel, the association determination module is specifically used to filter out the display timestamp associated with the current display timestamp in the display timestamp list within the radius of the normalization kernel; obtain a real-time frame spacing that matches the current video frame, and determine a predicted display timestamp associated with the current display timestamp based on the real-time frame spacing and the filtered display timestamp; and compose the display timestamp sub-list based on the display timestamp associated with the current display timestamp and the predicted display timestamp.

[0145] In one embodiment, the video code stream encoding data also includes a previous video frame, and the device also includes: an estimation module, used to obtain a current frame spacing based on a current decoding timestamp corresponding to the current video frame and a previous decoding timestamp corresponding to the previous video frame; and estimate the real-time frame spacing based on the current frame spacing.

[0146] Each module in the above-mentioned timestamp optimization device during video transcoding can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0147] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store buffer queue data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a timestamp optimization method in a video transcoding process is implemented.

[0148] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0149] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0150] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0151] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0153] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0154] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0155] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for optimizing timestamps during video transcoding, characterized in that: The method comprises: Read the video code stream encoding data of the source video; the video code stream encoding data includes the current video frame; Performing filtering processing on the display timestamp associated with the current video frame to obtain a filtered display timestamp corresponding to the current video frame; storing the filtered display timestamps in a buffer queue in chronological order; the buffer queue stores the filtered display timestamps sorted in a monotonically increasing order; After the video encoder outputs the transcoded current video frame, a decoding timestamp corresponding to the transcoded current video frame is determined in the buffer queue.

2. The method according to claim 1, characterized in that The determining, in the buffer queue, a decoding timestamp corresponding to the transcoded current video frame comprises: Taking out the first filtered display timestamp from the buffer queue; The first filtered display timestamp is used as the decoding timestamp corresponding to the transcoded current video frame.

3. The method according to claim 1, characterized in that There are multiple display timestamps associated with the current video frame, and filtering the display timestamps associated with the current video frame to obtain a filtered display timestamp corresponding to the current video frame includes: According to a preset normalization kernel, filtering is performed on the multiple display timestamps associated with the current video frame to obtain a filtered display timestamp corresponding to the current video frame.

4. The method according to claim 3, characterized in that: The method further comprises: Obtaining a current display timestamp corresponding to the current video frame, and caching the current display timestamp into a display timestamp list corresponding to the source video; According to the radius of the preset normalized kernel, a display timestamp sub-list associated with the current display timestamp is screened out from the display timestamp list; the display timestamp sub-list is centered on the current display timestamp and includes display timestamps before and after the current display timestamp; the amount of data in the display timestamp sub-list matches the radius; The display timestamp in the display timestamp sublist associated with the current display timestamp is used as the display timestamp associated with the current video frame.

5. The method according to claim 4, characterized in that In a case where the amount of data in the display timestamp list does not match the radius of the normalization kernel, filtering out a display timestamp sub-list associated with the current display timestamp from the display timestamp list according to the preset radius of the normalization kernel includes: Within the radius of the normalized kernel, selecting a display timestamp associated with the current display timestamp from a display timestamp list; Acquire a real-time frame spacing that matches the current video frame, and determine a predicted display timestamp associated with the current display timestamp according to the real-time frame spacing and the filtered display timestamp; The display timestamp sub-list is composed according to the display timestamp associated with the current display timestamp and the predicted display timestamp.

6. The method according to claim 5, characterized in that The video code stream encoding data also includes a previous video frame, and obtaining a real-time frame spacing matching the current video frame includes: Obtaining a current frame spacing according to a current decoding timestamp corresponding to the current video frame and a previous decoding timestamp corresponding to the previous video frame; The real-time frame spacing is estimated according to the current frame spacing.

7. A timestamp optimization device in a video transcoding process, characterized in that: The device comprises: A reading module, used for reading the video code stream encoding data of the source video; the video code stream encoding data includes the current video frame; A filtering module, configured to filter the display timestamp associated with the current video frame to obtain a filtered display timestamp corresponding to the current video frame; A storage module, used for storing the filtered display timestamps into a buffer queue in chronological order; the buffer queue stores the filtered display timestamps sorted in a monotonically increasing order; The determination module is used to determine the decoding timestamp corresponding to the transcoded current video frame in the buffer queue after the video encoder outputs the transcoded current video frame.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images