Method and apparatus for processing video skip show

Abstract

The present disclosure relates to a method and device for processing video jump display, wherein the method comprises: obtaining a jump request for a video; matching a time axis position indicated by the jump request with time axis positions of each decoded video frame in a cache queue to obtain a matching result; if the jump direction is forward jump, the matching result indicates that the jump request does not hit the cache queue; or if the jump direction is backward jump, the matching result indicates that the jump request does not hit the cache queue, and the video frame of the time axis position indicated by the jump request and the video frame in the cache queue do not belong to the same picture group, performing a seek operation, and not performing the seek operation in other scenarios. The present disclosure perceives scenarios that need to perform the seek operation through the cache queue, improves the rationality and necessity of performing the seek operation, and further improves the response speed to the jump request, reduces the video picture freezing phenomenon, and improves the user experience.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a processing method and apparatus for video jump display. Background Technology

[0002] Users of video-related applications often encounter video playback or frame previews, during which they can trigger seek requests. For example, when editing videos using video editing applications, users frequently preview the edited content by seeking. These applications provide editing tracks where users can add video clips they want to edit. Users can then trigger seek requests via the timeline to jump to a specific video frame at that timeline location. However, the editing track may contain multiple video clips, and these clips may even overlap on the timeline. If a seek request is needed to jump to the overlapping area, multiple video frame images must be captured, rendered, and composited before being displayed.

[0003] In existing technologies, when responding to a redirect request, the application first checks the cache queue. If the cache queue is not found, it responds by clearing existing cached data and calling the relevant Application Programming Interface (API) to control the decoder to decode the video frame at the specified timeline position requested in the redirect request from the corresponding reference frame, obtaining the decoded video frame at the specified time position and rendering it for display. Using this existing method, the application takes a long time to return the decoded video frame. In some scenarios, video frame images need to be switched and displayed quickly, and this long processing time can cause severe video stuttering and a poor user experience. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a method and apparatus for video jump display.

[0005] Firstly, this disclosure provides a method for handling video redirection and display, including:

[0006] Obtain the jump request triggered on the timeline of the video preview; the jump request is used to request a jump from the first timeline position to the second timeline position;

[0007] The matching result is obtained by matching the second timeline position and each decoded video frame in the cache queue;

[0008] If the first or second condition is met, the cache queue is cleared, and the decoder is controlled to decode based on the second timeline position and store the decoded video frames into the cache queue; if the third or fourth condition is met, the cache queue is not cleared; the first condition is that the jump request indicates a forward jump, and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position does not belong to the same frame group as the cache queue; the third condition is that the cache queue includes the video frame at the second timeline position; the fourth condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position belongs to the same frame group as the cache queue;

[0009] Based on the conditions met and the jump mode corresponding to the jump request, read the video frame at the second timeline position or any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and render and display it.

[0010] In some embodiments, the step of reading a video frame at the second timeline position or any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and rendering and displaying it according to the satisfied conditions and the jump mode indicated by the jump request includes:

[0011] If the first condition or the second condition is met, and the jump mode is the precise mode, then the video frame at the second timeline position is read from the cache queue and rendered and displayed.

[0012] If the first condition or the second condition is met, and the jump mode is a non-precise mode, then any video frame within the allowable deviation range corresponding to the second timeline position is read from the cache queue and rendered and displayed.

[0013] If the third or fourth condition is met, the video frame at the second timeline position is read from the cache queue and rendered and displayed.

[0014] In some embodiments, before reading any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and rendering and displaying it, the method further includes:

[0015] Calculate the allowable deviation range corresponding to the second time axis position.

[0016] In some embodiments, calculating the allowable deviation range corresponding to the second time axis position includes:

[0017] A first time difference is calculated based on the second time axis position and the third time axis position, wherein the third time axis position is the time axis position of the video frame displayed in the previous jump request;

[0018] Along the jump direction, the time range on the time axis that is at a distance from the second time axis position that satisfies the first time difference value is determined as the allowable deviation range.

[0019] In some embodiments, the control decoder performs decoding based on the second time axis position, including:

[0020] If the jump direction is forward jump, then when the decoder is being controlled to decode, non-reference frames on the timeline that are before the allowable deviation range will be discarded.

[0021] In some embodiments, the method further includes:

[0022] The target time threshold is calculated based on the maximum number of video frames that can be cached forward in the cache queue and the playback duration of a single video frame.

[0023] The second time difference is calculated based on the timeline position of the video frame output by the decoder and the timeline position of the video frame read in response to the jump request.

[0024] The second time difference is compared with the target time threshold to obtain a comparison result. Based on the comparison result, the video frame output by the decoder is written into the buffer queue or the video frame output by the decoder is discarded.

[0025] In some embodiments, the method further includes:

[0026] If it is determined that the video frame at the second timeline position cannot be successfully obtained based on the number of decoded video frames, the maximum range of reference frames, the position of the second timeline, the scheduled playback time of the first video frame in the buffer queue, and the scheduled playback duration of a single video frame, then the first video frame in the buffer queue is read and rendered for display.

[0027] Secondly, this disclosure provides a video jump display processing device, including:

[0028] The acquisition module is used to acquire jump requests triggered on the timeline of the video preview; the jump request is used to request a jump from a first timeline position to a second timeline position;

[0029] The matching module is used to obtain the matching result by matching based on the position of the second time axis and each decoded video frame in the cache queue;

[0030] The processing module is configured to: clear the cache queue and control the decoder to decode based on the second timeline position and store the decoded video frames into the cache queue if a first or second condition is met; not clear the cache queue if a third or fourth condition is met; the first condition is that the jump request indicates a forward jump and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position does not belong to the same frame group as the cache queue; the third condition is that the cache queue includes the video frame at the second timeline position; the fourth condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position belongs to the same frame group as the cache queue; and, according to the met conditions and the jump mode corresponding to the jump request, read the video frame at the second timeline position or any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and render and display it;

[0031] The display module is used to display the corresponding video frames.

[0032] Thirdly, this disclosure provides an electronic device, including: a memory and a processor;

[0033] The memory is configured to store computer program instructions;

[0034] The processor is configured to execute the computer program instructions, causing the electronic device to implement the video jump display processing method as described in the first aspect.

[0035] Fourthly, this disclosure provides a readable storage medium, comprising: computer program instructions; at least one processor of an electronic device executes the computer program instructions, causing the electronic device to implement the video jump display processing method as described in the first aspect.

[0036] Fifthly, this disclosure provides a program product in which an electronic device runs the computer program product, enabling the electronic device to implement the video jump display processing method as described in the first aspect.

[0037] This disclosure provides a method and apparatus for video jump display processing. The method includes: acquiring a jump request for a video; matching the timeline position indicated by the jump request with the timeline positions of each decoded video frame in the cache queue to obtain a matching result; if the jump direction is forward and the matching result indicates that the jump request did not hit the cache queue; or, if the jump direction is backward, the matching result indicates that the jump request did not hit the cache queue, and the video frame at the timeline position indicated by the jump request does not belong to the same frame group as the video frames in the cache queue, then a seek operation is performed; otherwise, a seek operation is not performed. This disclosure uses the cache queue to detect scenarios where a seek operation is actually required, improving the rationality and necessity of performing the seek operation, thereby improving the response speed to jump requests, reducing video stuttering, and enhancing the user experience. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0039] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A schematic diagram of the framework of the application provided in this disclosure;

[0041] Figure 2 A flowchart of a video jump display processing method provided in this embodiment of the disclosure;

[0042] Figure 3 This is a schematic diagram illustrating the relationship between the cache queue and the timeline provided in an embodiment of the present disclosure;

[0043] Figure 4 This is a schematic diagram illustrating a scenario where an application responds to a redirect request in the traditional way.

[0044] Figure 5 This is an illustrative diagram illustrating a scenario where a redirection request is responded to based on deviations from the permissible range, as demonstrated in this disclosure.

[0045] Figure 6 A schematic diagram illustrating the frame dropping strategy provided in an embodiment of this disclosure;

[0046] Figure 7 A flowchart of a video jump display processing method provided in this embodiment of the disclosure;

[0047] Figure 8This is a schematic diagram illustrating the relationship between the target time threshold and the cache queue provided in this disclosure;

[0048] Figure 9 A flowchart of a video jump display processing method provided in this embodiment of the disclosure;

[0049] Figure 10 A structural diagram of the processing device for video jump display provided in this disclosure. Detailed Implementation

[0050] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0051] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0052] Definitions of related terms:

[0053] Seek: In video preview, the process of jumping from one video frame at one timeline position to another is called a seek. The jump request disclosed herein can also be understood as a seek request.

[0054] Group of Pictures (GOP): A GOP is a group of consecutive frames. The first frame of a GOP must be an I-frame (I-frame) to ensure that the GOP can be decoded independently without referencing other frames. MPEG encoding divides frames into three types: I, P, and B. I is an internally coded frame, P is a forward-predicted frame, and B is a bidirectional interpolated frame. An I-frame is a keyframe and can be understood as a complete video frame. P and B frames record changes relative to the I-frame; a P-frame indicates the difference from the previous frame, and a B-frame indicates the difference between consecutive frames. The I-frame is the reference frame for decoding subsequent P and B frames. Without an I-frame, P and B frames cannot be decoded. The length of a GOP is the distance between two adjacent I-frames. For example, if the video frame sequence is: I, P, B, B, B, I, P, B, B, B, then the length of a GOP is 5, meaning one GOP contains 5 video frames.

[0055] For ease of description, the operation of clearing the existing cache and calling the relevant API to control the decoder to perform decoding from the reference frame corresponding to the specified timeline position indicated by the jump request will be referred to as the seek operation.

[0056] In traditional methods, when a jump request misses the cache queue, the application performs a seek operation. However, in some scenarios where fast video frame switching is desired, this traditional approach results in unnecessary computation, leading to prolonged processing time and severe video stuttering, resulting in a poor user experience. For example, if a jump request indicates a backward jump and misses the cache, and the decoded video frames in the cache queue belong to the same GOP as the video frame at the specified timeline position indicated by the jump request, a seek operation would be performed using the traditional method. This means that from the I-frame backward along the timeline to the last video frame in the cache queue, all video frames within that range are repeatedly decoded, resulting in significant computational time consumption and severe video stuttering.

[0057] In conclusion, analyzing the rationality and necessity of performing seek operations, and executing seek operations in the most necessary scenarios, is of great significance for improving the processing efficiency of video screen switching and display and solving the problem of video screen stuttering.

[0058] Based on this, the present disclosure provides a method and apparatus for video jump display. The method uses a cache queue to detect scenarios where a seek operation needs to be actually performed, thereby improving the rationality and necessity of performing the seek operation, thus improving the application's response speed to jump requests, reducing video stuttering, and enhancing the user experience.

[0059] For example, the video jump display processing method provided in this disclosure is executed by an electronic device. The electronic device may be a tablet computer, a mobile phone (such as a foldable screen phone, a large screen phone, etc.), a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart TV, a smart screen, a high-definition TV, a 4K TV, a smart speaker, a smart projector, and other Internet of Things (IoT) devices. This disclosure does not impose any restrictions on the specific type of electronic device.

[0060] Figure 1 A schematic diagram of the application framework provided in this disclosure; please refer to Figure 1As shown, the application includes a business layer, a middleware layer, and a decoding layer. The application can be, but is not limited to, a video playback application or a video editing application. In this disclosure, the business layer is mainly used to obtain jump requests triggered on the video preview timeline and pass the jump requests to the middleware layer, obtain the decoded video frames returned by the middleware layer in response to the jump requests, and render and display them. The middleware layer is mainly used to determine whether a seek operation needs to be performed based on information such as the specified timeline position indicated by the jump request, the timeline position of the decoded video frames in the cache queue, and the jump direction. If it is determined that a seek operation is needed, the relevant API is called to perform the seek operation. The decoding layer is mainly used to respond to the instructions of the middleware layer and perform decoding processing on the video frames in the decoding queue.

[0061] The following embodiments use an electronic device as an example, and the electronic device has a video-related application installed. The application uses, for example, Figure 1 Using the framework shown as an example, the method provided in this disclosure will be described in detail.

[0062] Figure 2 This is a flowchart illustrating a video transition and display method according to an embodiment of this disclosure. Please refer to [link / reference]. Figure 2 As shown, the method in this embodiment may include:

[0063] S101. Obtain the jump request triggered on the timeline of the video preview; the jump request is used to request a jump from the first timeline position to the second timeline position.

[0064] The application's business layer can display an interactive interface on the electronic device's screen. Users can trigger jump requests by touching or operating external devices connected to the electronic device (such as a mouse) to interact with the timeline displayed on the interface, such as video playback timelines, preview timelines, and other components. The application's business layer determines the position of the second timeline, the jump mode, and other information based on the user's interaction with the timeline, and generates a jump request based on the determined information. Jump modes will be described in detail later.

[0065] It should be noted that this disclosure does not limit the way users trigger video jumps. In addition to the methods shown above, users can also trigger jump requests through the full-screen drag function, media drag function, thumbnail drag function, etc. provided by the application.

[0066] S102. Match the video frames in the second timeline and the decoded video frames in the buffer queue to obtain the matching result.

[0067] A buffer queue is used to buffer decoded video frames output by the decoder. For example, see [link to documentation]. Figure 3 As shown, a certain video frame is used as the reference video frame, and the position of the reference video frame on the timeline is as follows. Figure 3 As shown by the black vertical lines, the buffer queue can include forward buffers and backward buffers corresponding to the base video frame. The forward buffer includes video frames whose timeline positions are lower than the base video frame's timeline position, while the backward buffer includes video frames whose timeline positions are higher than the base video frame's timeline position. The forward and backward buffers can satisfy a preset ratio, for example, the forward buffer occupies 0.7% of the buffer queue, and the backward buffer occupies 0.3%.

[0068] In this disclosure, the application's middleware layer internally detects and determines whether a seek operation needs to be performed, breaking the traditional reliance on the business layer. The module in the middleware layer that can quickly and accurately detect whether a seek operation needs to be performed is the cache queue. Therefore, when the application's middleware layer receives a jump request, it will query the cache queue based on the second timeline position indicated by the jump request to obtain a matching result.

[0069] The purpose of querying the cache queue is to obtain two types of information: 1. Whether the jump request hits the cache queue, that is, whether the cache queue includes the video frame at the second timeline position; 2. Whether the video frame at the second timeline position and each decoded video frame in the cache queue belong to the same frame group.

[0070] The cache queue can determine whether a jump request hits the cache queue by comparing the second timeline position with the timeline position of the first video frame in the cache queue, and by comparing the second timeline position with the timeline position of the last video frame in the cache queue. If it hits, it can simultaneously determine whether the video frame at the second timeline position and each decoded video frame in the cache queue belong to the same GOP; if it misses, it obtains the timeline position of the corresponding I-frame and the time interval of the I-frame, determines whether the second timeline position is greater than the timeline position of the I-frame and less than the timeline position of the next I-frame to determine whether the second timeline position belongs to the GOP, and determines whether the cache queue belongs to the GOP, thus obtaining the information in the second aspect mentioned above.

[0071] S103. If either the first or second condition is met, the cache queue is cleared, and the decoder is controlled to decode based on the second timeline position and store the decoded video frames into the cache queue. The first condition is that the jump request indicates a forward jump, and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position does not belong to the same frame group as the cache queue.

[0072] S104. If the third or fourth condition is met, the cache queue is not cleared. The third condition is: the cache queue includes the video frame at the second timeline position; the fourth condition is: the jump direction is backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position belongs to the same frame group as the cache queue.

[0073] The application's middleware layer determines whether a seek operation is required based on the redirection direction indicated by the redirection request passed from the business layer and the matching results returned by the cache queue. Based on the combination of redirection direction and matching results, the scenarios can be categorized as follows:

[0074] Scenario 1: Forward jump, jump request misses the cache queue, the video frame with the first timestamp and each decoded video frame in the cache queue belong to the same GOP;

[0075] Scenario 2: Forward jump, jump request misses cache queue, video frame with first timestamp does not belong to the same GOP as the decoded video frames in the cache queue;

[0076] Scenario 3: Backward jump, jump request misses the cache queue, video frame with the first timestamp does not belong to the same GOP as the decoded video frames in the cache queue;

[0077] Scenario 4: Backward jump, jump request misses the cache queue, the video frame with the first timestamp belongs to the same GOP as each decoded video frame in the cache queue;

[0078] Scenario 5: Forward jump, jump request hits the cache queue;

[0079] Scenario 6: Backward redirection, redirection request hits the cache queue.

[0080] If the application's intermediate layer determines that the current scenario is any of scenario 1 to scenario 3, then it determines that a seek operation needs to be performed; if the intermediate layer determines that the current scenario is any of scenario 4 to scenario 6, then it determines that a seek operation does not need to be performed.

[0081] After determining that a seek operation is needed, the intermediate layer (buffer queue) sends a jump request and an indication signal to the decoding layer. This indication signal indicates that the buffer queue is currently in a frame-missing state; it can also indicate a frame-missing frame forward or backward. Based on the second timeline indicated by the jump request and the frame-missing direction, the decoding layer determines the decoding start frame and begins decoding processing from that frame, writing the decoded video frames output by the decoder into the buffer queue. The decoding start frame is typically the first video frame (I-frame) of the Group of Pictures (GOP), serving as the reference frame for decoding.

[0082] S105. Based on the conditions met and the jump mode corresponding to the jump request, read the video frame at the second timeline position or any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and render and display it.

[0083] The application's middleware layer can determine whether the video frame to be returned is a precise frame or an inaccurate frame based on the conditions met by the jump request and the jump mode. It then reads the corresponding video frame from the cache queue and sends it to the business layer, where it renders and displays the video, thus successfully achieving the screen jump display.

[0084] If either the first or second condition is met, and the jump mode is precise, then the video frame at the second timeline position is read from the cache queue and rendered. If either the first or second condition is met, and the jump mode is non-precise, then any video frame within the allowable deviation range corresponding to the second timeline position is read from the cache queue and rendered. If the third condition is met, then the video frame at the second timeline position is directly read from the cache queue and rendered. If the fourth condition is met, then the current decoding task is not interrupted, and the decoder continues to decode the video frames after the last video frame in the cache queue along the timeline, and writes the decoded data of the decoded video frames into the cache queue. Then, the video frame at the second timeline position is read from the cache queue and rendered.

[0085] The method in this embodiment uses a cache queue to detect scenarios where a seek operation actually needs to be performed, thereby improving the rationality and necessity of performing the seek operation. It then responds to the jump request in a timely manner, returns the target video frame that meets the jump request for rendering and display, reduces video stuttering, and improves the user experience.

[0086] Next, we will provide a detailed explanation of how to read video frames when the jump mode is in non-precise mode.

[0087] In this disclosure, the application's middleware layer provides two seek modes for the business layer to choose from: 1. Precise seek mode; 2. Sliding seek mode, also known as non-precise seek mode. The business layer can determine which seek mode to use based on the business scenario. For example, in frame-by-frame seek or fixed-frame seek scenarios, the precise seek mode must be used; if the seek is triggered by preview axis functionality, full-screen drag functionality, or material drag functionality, and high smoothness is required, the non-precise seek mode can be used.

[0088] After the business layer determines which seek mode to use based on the business scenario, it can add mode indication information to the jump request so that the middle layer can determine the seek mode and thus determine which frame or time range of video frame on the timeline needs to be returned. It should be understood that if the seek mode is a precise seek mode, then the video frame at the second timeline position needs to be returned, and video frames at other timeline positions cannot meet the scenario requirements; if the seek mode is a non-precise seek mode, the returned video frame can deviate from the second timeline position, prioritizing the smoothness of the video.

[0089] In non-precise seek mode, the returned video frames can be understood as non-precise frames. Since backward seek can return any frame after the currently displayed video frame, it ensures smooth playback. However, in forward seek scenarios, the appropriateness of the selection of non-precise frames significantly impacts seek performance. For example, refer to... Figure 4 As shown, assuming four seeks are performed forward along the timeline, the video frame returned by the first seek is located between the timeline positions of the first and second seeks, and the video frame returned by the second seek is located before the timeline position of the fourth seek. Since the strategy for selecting inaccurate frames is to choose the nearest possible video frame, to prevent the image from jumping back and forth, the third and fourth seeks will not return any video frames, and the video will always display the video frame returned by the second seek. Therefore, in forward seek scenarios, how to select inaccurate frames is the key to implementing inaccurate seek.

[0090] To improve the performance of inaccurate seeking in forward seeking scenarios, this disclosure predicts the timeline position specified in the next jump request and determines the allowable deviation range for selecting inaccurate frames using the second timeline position and the timeline position specified in the next jump request. As one possible implementation, calculating the allowable deviation range can be achieved through the following steps:

[0091] Step a1: Calculate the first time difference based on the second timeline position and the third timeline position of the video frame displayed in response to the previous jump request. This can be expressed by the formula: Δt = t1 - t0, where Δt represents the first time difference, t1 represents the second timeline position, and t0 represents the third timeline position. It should be understood that the jump direction indicated by the previous jump request was a forward jump.

[0092] Step a2: Following the jump direction indicated by this jump request, determine the allowable deviation range as the time interval between the second time axis position and the second time axis position that satisfies the first time difference value. This can be expressed by the formula: t2 = Δt + t1, where t2 represents one end point of the allowable deviation range, which can be denoted as the fourth time axis position. The time interval between t2 and t1 on the time axis belongs to the allowable deviation range.

[0093] For example, assuming this jump request indicates a forward jump, the positions of t2 and t1 on the timeline can be referenced. Figure 5 As shown, any video frame between t2 and t1 can be an inaccurate frame.

[0094] contrast Figure 4 and Figure 5 As can be seen from the embodiments shown, when it is determined that a seek operation needs to be performed and the jump mode is determined to be a non-precise seek mode, this disclosure can obtain the corresponding deviation tolerance range by predicting the next jump request. Taking non-precise frames within the deviation tolerance range can improve the seek hit rate and avoid the video screen jumping back and forth.

[0095] It should be noted that non-precise seeking can also be achieved by issuing a deviation tolerance range through the business layer. Alternatively, users can pre-configure a first time difference, so that the intermediate layer can determine the deviation tolerance range based on the pre-configured first time difference. Of course, there are other ways to determine the deviation tolerance range, not limited to the methods shown in the examples above.

[0096] by Figure 5 Taking the example shown, in a typical forward jump scenario, the decoder is triggered to perform decoding processing from the I-frame. Due to the decoding order, the buffer queue usually writes the decoded data of the video frame at position t2 first. The intermediate layer will also prioritize detecting the video frame at position t2 in the buffer queue. Since the video frame at position t2 meets the deviation tolerance range, it will be returned as the video frame that satisfies the jump request. It should be understood that if multiple video frames within the deviation tolerance range are detected in the buffer queue, the video frame closest to the second timeline position on the timeline will be returned as the target video frame.

[0097] To further improve seek performance, this disclosure optimizes the implementation through a frame dropping strategy. In the traditional approach, the decoder directly discards non-reference frames preceding the timeline position indicated by the jump request. However, direct discarding can lead to errors in calculating the predetermined playback duration of video frames in the decoding queue. The predetermined playback duration of a video frame is calculated based on the predetermined playback times of two adjacent frames in the decoding queue. If some non-reference frames are discarded, the predetermined playback duration of the video frame will be calculated incorrectly.

[0098] This disclosure optimizes the frame dropping strategy; see [link / reference] Figure 6 As shown, when the decoder performs decoding based on the jump request and indication signal, it discards non-reference frames located between the I-frame and the video frame corresponding to the prediction point (such as position t2 in the previous embodiment). This reduces the number of video frames that need to be decoded, decreases the decoding computation, and improves seek performance.

[0099] After the application returns a video frame, it needs to update a time threshold. Decoded video frames shorter than this threshold are discarded, while those longer than or equal to the threshold are written to a buffer queue to update the queue. Traditionally, this time threshold is calculated based on the number of preceding and following frames in the buffer queue and the predetermined playback time (PTS) of each video frame. However, when this traditional method is applied to scenarios involving forward jumps, a large number of decoded video frames cannot enter the buffer queue.

[0100] Figure 7 This is a flowchart illustrating a video transition and display method according to an embodiment of this disclosure. Please refer to [link / reference]. Figure 7 As shown, the method in this embodiment includes:

[0101] S201. Calculate the target time threshold based on the maximum number of video frames that can be cached forward in the cache queue and the playback duration of a single video frame.

[0102] Typically, the length of the cache queue, the ratio of forward caching to backward caching, etc. are preset. The maximum number of video frames that can be cached forward is determined based on the proportion of forward caching to the total length of the cache queue and the length of the cache queue. Then, the maximum number of video frames that can be cached forward is multiplied by the predetermined single-frame playback duration to obtain the target time threshold.

[0103] For example, assuming the forward cache can cache up to 20 frames, and the playback duration of each frame is 33 milliseconds, the target time threshold = 20 * 0.033 = 0.66 seconds.

[0104] S202. Calculate the second time difference value based on the time axis position of the video frame output by the decoder and the time axis position of the video frame read in response to the jump request.

[0105] S203. Compare the second time difference with the target time threshold to obtain the comparison result. Based on the comparison result, write the video frame output by the decoder into the buffer queue or discard the video frame output by the decoder.

[0106] If the comparison result indicates that the second time difference is less than the target time threshold, the decoded video frame output by the decoder is written into the buffer queue; if the comparison result indicates that the second time difference is greater than or equal to the target time threshold, the decoded video frame output by the decoder is discarded.

[0107] This embodiment calculates the target time threshold by the number of video frames that can be cached in the forward cache, and updates the cache queue so that there are enough decoded video frames in the forward cache, thereby ensuring that the forward cache can meet the needs of the next forward jump as much as possible.

[0108] In a specific embodiment, assume that the application generates four consecutive redirect requests based on the user's continuous operations, and all four redirect requests are forward redirects, with each redirect returning t. a t b t c t d The video frames at four time points are processed after each video frame is returned. Figure 8 The illustrated embodiment determines the target time threshold and updates the forward cache in the cache queue.

[0109] Because decoding anomalies may occur during the decoding process of the original video data, and when the jump mode is in precise seek mode, it is certain that precise frames may not be obtained after the seek operation is performed.

[0110] Figure 9 A flowchart illustrating a video transition display method provided in another embodiment of this disclosure. Please refer to [link / reference]. Figure 9 As shown, the method in this embodiment includes:

[0111] S301. Obtain the jump request triggered on the timeline of the video preview; the jump request is used to request a jump from the first timeline position to the second timeline position.

[0112] S302. Match the video frames in the second timeline and the decoded video frames in the buffer queue to obtain the matching result.

[0113] S303. If the first condition or the second condition is met, clear the cache queue and control the decoder to decode based on the second time axis position and store the decoded video frames into the cache queue.

[0114] In this embodiment, steps S301 to S303 are... Figure 2 Steps S101 to S103 in the illustrated embodiment are similar and can be referred to. Figure 2 The detailed description of the illustrated embodiments will not be repeated here for the sake of brevity.

[0115] S304. Based on the number of decoded video frames, the maximum range of reference frames, the position of the second timeline, the scheduled playback time of the first video frame in the buffer queue, and the scheduled playback duration of a single video frame, determine whether the video frame at the second timeline position can be successfully obtained.

[0116] If the redirection mode is determined to be precise mode based on the redirection request, it is then determined whether a precise frame can be successfully obtained. For example, based on the above information, it is determined whether the following conditions are currently met:

[0117] Condition 1: The number of decoded video frames in the decoding queue is greater than the maximum range of reference frames;

[0118] Condition 2: The first timestamp is less than the scheduled playback time of the first video frame in the cache queue;

[0119] Condition 3: The time interval between the scheduled playback time and the first timestamp of the first video frame in the cache queue is less than the scheduled playback duration of a single video frame;

[0120] Condition 4: The scheduled playback time and first timestamp of the first video frame in the cache queue are less than the preset time offset, which is equal to the product of the maximum range of the reference frame and the scheduled playback duration of a single video frame.

[0121] If conditions 1 to 4 above are met simultaneously, it is determined that the video frame at the second timeline position cannot be successfully obtained to achieve precise seek, and step S305 is executed. The inability to read the precise frame corresponding to the jump request may be due to an error in the maximum range of the reference frame provided by the video, leading to decoding errors.

[0122] It should be noted that in precise seek mode, if any of the above conditions are not met, it means that the application's intermediate layer can implement precise seek.

[0123] S305: Read the first video frame from the cache queue and render and display it.

[0124] The application's middleware reads the decoded data of the first video frame from the cache queue and sends it to the business layer, which then renders and displays it, thus successfully achieving the screen transition.

[0125] This embodiment is essentially a fault-tolerant processing mechanism for precise seek scenarios. In the case of forward jump, when it is determined that a precise frame cannot be obtained, the video frame closest to the second timeline position in the cache queue is returned as the precise frame to ensure the smoothness of video jump.

[0126] Figure 10 This is a structural diagram of a video jump display processing device provided according to an embodiment of this disclosure. Please refer to... Figure 10As shown, the device 1000 provided in this embodiment includes:

[0127] The acquisition module 1001 is used to acquire a jump request triggered on the timeline of the video preview; the jump request is used to request a jump from a first timeline position to a second timeline position;

[0128] Matching module 1002 is used to obtain a matching result by matching according to the second time axis position and each decoded video frame in the buffer queue;

[0129] The processing module 1003 is configured to: if a first condition or a second condition is met, clear the cache queue and control the decoder to decode based on the second timeline position and store the decoded video frames into the cache queue; if a third condition or a fourth condition is met, not clear the cache queue; the first condition is that the jump request indicates a forward jump and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position does not belong to the same frame group as the cache queue; the third condition is that the cache queue includes the video frame at the second timeline position; the fourth condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position belongs to the same frame group as the cache queue; and, according to the met conditions and the jump mode corresponding to the jump request, read the video frame at the second timeline position or any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and render and display it;

[0130] Display module 1004 is used to display the corresponding video frames.

[0131] In some embodiments, the processing module 1003 is specifically configured to: if the first condition or the second condition is met, and the jump mode is a precise mode, then read the video frame at the second timeline position from the cache queue and render and display it; if the first condition or the second condition is met, and the jump mode is a non-precise mode, then read any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and render and display it; if the third condition or the fourth condition is met, then read the video frame at the second timeline position from the cache queue and render and display it.

[0132] In some embodiments, before reading any video frame within the deviation tolerance range corresponding to the second time axis position from the cache queue and rendering and displaying it, the processing module 1003 is further configured to: calculate the deviation tolerance range corresponding to the second time axis position.

[0133] In some embodiments, the processing module 1003 is specifically configured to: calculate a first time difference based on the second time axis position and the third time axis position, wherein the third time axis position is the time axis position of the video frame displayed in the previous jump request; and determine, along the jump direction, the time range on the time axis from the second time axis position as the starting point, where the distance from the second time axis position satisfies the first time difference as the allowable deviation range.

[0134] In some embodiments, the processing module 1003 is specifically configured to: if the jump direction is a forward jump, control the decoder to discard non-reference frames on the time axis that are before the allowable deviation range when performing decoding.

[0135] In some embodiments, the processing module 1003 is further configured to: calculate a target time threshold based on the maximum number of video frames that can be cached forward in the cache queue and the playback duration of a single video frame; calculate a second time difference based on the timeline position of the video frame output by the decoder and the timeline position of the video frame read in response to the jump request; compare the second time difference with the target time threshold to obtain a comparison result; and write the video frame output by the decoder into the cache queue or discard the video frame output by the decoder based on the comparison result.

[0136] In some embodiments, the processing module 1003 is further configured to: if it is determined that the video frame at the second timeline position cannot be successfully obtained based on the number of decoded video frames, the maximum range of reference frames, the second timeline position, the predetermined playback time of the first video frame in the buffer queue, and the predetermined playback duration of a single video frame, then read the first video frame in the buffer queue and render and display it.

[0137] The apparatus provided in this embodiment can be used to implement the technical solutions of any of the foregoing method embodiments. The implementation principles and technical effects are similar, and can be referred to in the detailed description of the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0138] For example, this disclosure provides an electronic device, including: one or more processors; a memory; and one or more computer programs; wherein the one or more computer programs are stored in the memory; and when the one or more processors execute the one or more computer programs, the electronic device enables the video jump display processing method of the foregoing embodiments.

[0139] For example, this disclosure provides a chip system applied to an electronic device including a memory and a sensor; the chip system includes: a processor; and the processing method for video jump display in the preceding embodiments is executed by the processor.

[0140] For example, this disclosure provides a computer-readable storage medium having a computer program stored thereon, the computer program being processed by a processor to cause an electronic device to execute the video jump display processing method described in the preceding embodiments.

[0141] For example, this disclosure provides a computer program product that, when run on a computer, causes the computer to execute the video jump display processing method described in the preceding embodiments.

[0142] In the above embodiments, all or part of the functionality can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented wholly or partially in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or functionality according to the embodiments of this disclosure is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0143] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing video jump displays, characterized in that, include: Get the jump request triggered on the video preview timeline; The jump request is used to request a jump from the first time axis position to the second time axis position; The matching result is obtained by matching the second timeline position and each decoded video frame in the cache queue; If the first or second condition is met, the cache queue is cleared, and the decoder is controlled to decode based on the second timeline position and store the decoded video frames into the cache queue; if the third or fourth condition is met, the cache queue is not cleared. The first condition is that the jump request indicates a forward jump, and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position does not belong to the same frame group as the cache queue; the third condition is that the cache queue includes the video frame at the second timeline position; the fourth condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position belongs to the same frame group as the cache queue; Based on the conditions met and the jump mode corresponding to the jump request, read the video frame at the second timeline position or any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and render and display it; Before reading any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and rendering and displaying it, the method further includes: A first time difference is calculated based on the second time axis position and the third time axis position, wherein the third time axis position is the time axis position of the video frame displayed in the previous jump request; Along the jump direction, the time range on the time axis that is at a distance from the second time axis position that satisfies the first time difference value is determined as the allowable deviation range.

2. The method according to claim 1, characterized in that, The step of reading a video frame at the second timeline position or any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and rendering and displaying it according to the satisfied conditions and the jump mode indicated by the jump request includes: If the first condition or the second condition is met, and the jump mode is the precise mode, then the video frame at the second timeline position is read from the cache queue and rendered and displayed. If the first condition or the second condition is met, and the jump mode is a non-precise mode, then any video frame within the allowable deviation range corresponding to the second timeline position is read from the cache queue and rendered and displayed. If the third or fourth condition is met, the video frame at the second timeline position is read from the cache queue and rendered and displayed.

3. The method according to claim 1, characterized in that, The control decoder performs decoding based on the second time axis position, including: If the jump direction is forward jump, then when the decoder is being controlled to decode, non-reference frames on the timeline that are before the allowable deviation range will be discarded.

4. The method according to claim 1, characterized in that, The method further includes: The target time threshold is calculated based on the maximum number of video frames that can be cached forward in the cache queue and the playback duration of a single video frame. The second time difference is calculated based on the timeline position of the video frame output by the decoder and the timeline position of the video frame read in response to the jump request. The second time difference is compared with the target time threshold to obtain a comparison result. Based on the comparison result, the video frame output by the decoder is written into the buffer queue or the video frame output by the decoder is discarded.

5. The method according to claim 2, characterized in that, The method further includes: If it is determined that the video frame at the second timeline position cannot be successfully obtained based on the number of decoded video frames, the maximum range of reference frames, the position of the second timeline, the scheduled playback time of the first video frame in the buffer queue, and the scheduled playback duration of a single video frame, then the first video frame in the buffer queue is read and rendered for display.

6. A processing device for video jump display, characterized in that, include: The acquisition module is used to acquire jump requests triggered on the timeline of the video preview; The jump request is used to request a jump from the first time axis position to the second time axis position; The matching module is used to obtain the matching result by matching based on the position of the second time axis and each decoded video frame in the cache queue; The processing module is configured to, if the first or second condition is met, clear the cache queue, control the decoder to perform decoding based on the second timeline position, and store the decoded video frames into the cache queue; if the third or fourth condition is met, not clear the cache queue. The first condition is that the jump request indicates a forward jump, and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position does not belong to the same frame group as the cache queue; the third condition is that the cache queue includes the video frame at the second timeline position; the fourth condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position belongs to the same frame group as the cache queue; Furthermore, based on the conditions met and the jump mode corresponding to the jump request, the video frame at the second timeline position or any video frame within the allowable deviation range corresponding to the second timeline position is read from the cache queue and rendered and displayed; The display module is used to display the corresponding video frames; Before reading any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and rendering and displaying it, the processing module is further configured to: A first time difference is calculated based on the second time axis position and the third time axis position, wherein the third time axis position is the time axis position of the video frame displayed in the previous jump request; Along the jump direction, the time range on the time axis that is at a distance from the second time axis position that satisfies the first time difference value is determined as the allowable deviation range.

7. An electronic device, characterized in that, include: Memory and processor; The memory is configured to store computer program instructions; The processor is configured to execute the computer program instructions, causing the electronic device to implement the video jump display processing method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, include: Computer program instructions; At least one processor of the electronic device executes the computer program instructions, causing the electronic device to implement the video jump display processing method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The electronic device runs the computer program product, enabling the electronic device to implement the video jump display processing method as described in any one of claims 1 to 5.

Images