Video seek display processing method and apparatus

Through cache queue perception and optimized decoding process, the video screen lag caused by slow response to jump requests in video processing technology is solved, and the user experience and video processing efficiency are improved.

WO2024235329A1PCT designated stage expired Publication Date: 2024-11-21BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/094065
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-18
Filing Date
2024-05-17
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

When responding to jump requests, existing video processing technologies take a long time to cause the video screen to stutter and the user experience is poor, especially when video frames need to be quickly switched to display.

Method used

Through cache queue perception, the scenarios where seek operations need to be performed will be improved, the rationality and necessity of seek operations will be optimized, the decoding and rendering process will be reduced, and the deviation tolerance range and frame drop strategy will be adopted to improve the response speed.

Benefits of technology

It improves the response speed of video jump requests, reduces the phenomenon of video screen lag, improves user experience, and ensures the processing efficiency of video screen switching display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024094065_21112024_PF_FP_ABST
    Figure CN2024094065_21112024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to a video seek display processing method and apparatus. The method comprises: acquiring a seek request for a video; matching a time axis position indicated by the seek request with a time axis position of each decoded video frame in a cache queue to obtain a matching result; and when a seek direction indicates forward seek and the matching result indicates that the seek request does not hit the cache queue, or the seek direction indicates backward seek, the matching result indicates that the seek request does not hit the cache queue, and a video frame at the time axis position indicated by the seek request and the video frames in the cache queue do not belong to a same picture group, performing a seek operation, and not performing a seek operation in other scenarios. According to the present disclosure, a scenario in which a seek operation needs to be truly executed is perceived by means of a cache queue, thereby improving the rationality and necessity of execution of a seek operation, then increasing the speed of response to a seek request, reducing the video picture lag phenomenon, and improving the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Video jump display processing method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese invention patent application number 202310564980.7, entitled “Processing method and device for video jump display” and filed on May 18, 2023, and the entire application is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of video processing technology, and in particular to a method and device for processing video jump display. Background Art

[0004] When users use video-related applications, they often encounter video playback or video frame preview, during which users can trigger a jump (seek) request. For example, when users use a video editing application to edit a video, they often preview the editing effect by jumping. The application provides an editing track, and users can add the video track segment they want to edit to the editing track. Users can trigger a jump request through the timeline to jump to the video frame at the specified timeline position. Among them, multiple video track segments may be imported into the editing track, and even multiple video track segments may overlap on the timeline. If you jump to a position where multiple video track segments overlap, you need to obtain multiple video frame images and render and synthesize them before displaying them on the screen.

[0005] In the prior art, in response to a jump request, the application will first query the cache queue. If the cache queue is not hit, it will respond to the jump request to clear the existing cache data and call the relevant application programming interface (API) to control the decoder to decode the corresponding reference frame based on the specified timeline position requested by the jump request to obtain the decoded video frame at the specified time position and render it on the screen for display. Using the above-mentioned existing method, it takes a long time for the application to return the decoded video frame. In some scenarios, the video frame image needs to be switched quickly for display. The long time consumption will cause serious video image freezes and a poor user experience.

[0006] Summary of the Invention

[0007] In order to solve the above technical problems, the present disclosure provides a method and device for processing video jump display.

[0008] In a first aspect, the present disclosure provides a method for processing video jump display, comprising:

[0009] Obtaining a jump request triggered on the timeline of the video preview; the jump request is used to request jumping from a first timeline position to a second timeline position;

[0010] Matching is performed according to the second timeline position and each decoded video frame in the cache queue to obtain a matching result;

[0011] If the first condition or the second condition is met, the cache queue is cleared, and the decoder is controlled to perform decoding based on the second timeline position and store the decoded video frame into the cache queue; if the third condition or the fourth condition is met, the cache queue is not cleared; the first condition is that the jump request indicates a forward jump, and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position and the cache queue do not belong to the same picture group; the third condition is that the cache queue includes the video frame at the second timeline position; the fourth condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position and the cache queue belong to the same picture group;

[0012] According to the satisfied conditions and the jump mode corresponding to the jump request, the video frame at the second timeline position or any video frame within the deviation tolerance range corresponding to the second timeline position is read from the cache queue and rendered for display.

[0013] In some embodiments, reading the video frame at the second timeline position or any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and rendering and displaying the video frame according to the satisfied condition and the jump mode indicated by the jump request includes:

[0014] If the first condition or the second condition is met and the jump mode is the precise mode, the video frame at the second timeline position is read from the cache queue and rendered for display;

[0015] If the first condition or the second condition is met and the jump mode is the non-precise mode, any video frame within the deviation tolerance range corresponding to the second timeline position is read from the cache queue and rendered for display;

[0016] If the third condition or the fourth condition is met, the video frame at the second timeline position is read from the cache queue and rendered for display.

[0017] In some embodiments, before reading any video frame within the deviation tolerance range corresponding to the second timeline position from the cache queue and rendering and displaying it, the method further includes:

[0018] Calculate the deviation tolerance corresponding to the second time axis position.

[0019] In some embodiments, calculating the deviation tolerance corresponding to the second time axis position includes:

[0020] Calculating a first time difference according to the second timeline position and a third timeline position, wherein the third timeline position is the timeline position of the video frame displayed in the previous jump request;

[0021] Along the jump direction, a time range starting from the second time axis position and having a distance from the second time axis position that satisfies the first time difference is determined as the deviation tolerance range.

[0022] In some embodiments, controlling the decoder to perform decoding based on the second timeline position includes:

[0023] If the jump direction is forward jump, the decoder is controlled to discard non-reference frames located before the deviation tolerance range on the time axis during decoding.

[0024] In some embodiments, the method further comprises:

[0025] Calculating a target time threshold based on a maximum number of video frames that can be cached forward by the cache queue and a playback duration of a single video frame;

[0026] Calculating a second time difference value according to the time axis position of the video frame output by the decoder and the time axis position of the video frame read in response to the jump request;

[0027] The second time difference is compared with the target time threshold to obtain a comparison result, and the video frame output by the decoder is written into the cache queue or the video frame output by the decoder is discarded according to the comparison result.

[0028] In some embodiments, the method further comprises:

[0029] If it is determined that the video frame at the second timeline position cannot be successfully obtained based on the number of decoded video frames, the maximum range of reference frames, the second timeline position, the scheduled playback time of the first video frame in the cache queue, and the scheduled playback duration of a single video frame, the first video frame in the cache queue is read and rendered for display.

[0030] In a second aspect, the present disclosure provides a video jump display processing device, comprising:

[0031] An acquisition module, configured to acquire a jump request triggered on a timeline of a video preview; the jump request is used to request a jump from a first timeline position to a second timeline position;

[0032] a matching module, configured to perform matching according to the second timeline position and each decoded video frame in the cache queue to obtain a matching result;

[0033] a processing module, configured to clear the cache queue if a first condition or a second condition is satisfied, and control the decoder to perform decoding based on the second timeline position and store the decoded video frame in the cache queue; and not clear the cache queue if a third condition or a fourth condition is satisfied; the first condition being that the jump request indicates a forward jump and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition being that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position and the cache queue do not belong to the same picture group; the third condition being that the cache queue includes the video frame at the second timeline position; the fourth condition being that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position and the cache queue belong to the same picture group; and, based on the satisfied conditions and the jump mode corresponding to the jump request, read the video frame at the second timeline position or any video frame within a deviation tolerance range corresponding to the second timeline position from the cache queue and render and display the frame;

[0034] The display module is used to display the corresponding video frame.

[0035] In a third aspect, the present disclosure provides an electronic device, comprising: a memory and a processor;

[0036] The memory is configured to store computer program instructions;

[0037] The processor is configured to execute the computer program instructions so that the electronic device implements the video jump display processing method as described in the first aspect.

[0038] In a fourth aspect, the present disclosure provides a readable storage medium, comprising: computer program instructions; at least one processor of an electronic device executes the computer program instructions, so that the electronic device implements the processing method for video jump display as described in the first aspect.

[0039] In a fifth aspect, the present disclosure provides a program product, and an electronic device runs the computer program product, so that the electronic device implements the processing method for video jump display as described in the first aspect.

[0040] The embodiment of the present disclosure provides a method and device for processing video jump display, wherein the method includes: obtaining a jump request for a video; matching the timeline position indicated by the jump request and the timeline position of each decoded video frame in the cache queue to obtain a matching result; if the jump direction is a forward jump, the matching result indicates that the jump request does not hit the cache queue; or, if the jump direction is a backward jump, the matching result indicates that the jump request does not hit the cache queue, and the video frame at the timeline position indicated by the jump request and the video frame in the cache queue do not belong to the same picture group, then a seek operation is performed, and no seek operation is performed in other scenarios. The present disclosure senses the scenarios where a seek operation actually needs to be performed through the cache queue, thereby improving the rationality and necessity of performing the seek operation, thereby improving the response speed to the jump request, reducing the video picture freeze phenomenon, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0042] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] FIG1 is a schematic diagram of the framework of the application provided by the present disclosure;

[0044] FIG2 is a flow chart of a method for processing video jump display provided by an embodiment of the present disclosure;

[0045] FIG3 is a schematic diagram of the relationship between the cache queue and the time axis provided by an embodiment of the present disclosure;

[0046] FIG4 is a schematic diagram of a scenario in which an application responds to a jump request in a traditional manner;

[0047] FIG5 is a schematic diagram of a scenario of responding to a jump request based on deviation from an allowable range, exemplarily shown in the present disclosure;

[0048] FIG6 is a schematic diagram of a frame loss strategy provided by an embodiment of the present disclosure;

[0049] FIG7 is a flowchart of a method for processing video jump display provided by an embodiment of the present disclosure;

[0050] FIG8 is a schematic diagram showing the relationship between the target time threshold and the cache queue provided by the present disclosure;

[0051] FIG9 is a flowchart of a method for processing video jump display provided by an embodiment of the present disclosure;

[0052] FIG10 is a structural diagram of a processing device for video jump display provided by the present disclosure. DETAILED DESCRIPTION

[0053] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0055] Definitions of relevant terms:

[0056] Seek: During video preview, the process of jumping from a video frame at one timeline position to a video frame at another timeline position is called seek. The jump request disclosed in this disclosure can also be understood as a seek request.

[0057] Group of Pictures (GOP): A GOP is a set of consecutive pictures. The first picture in a GOP must be an I-frame. This ensures that the GOP can be decoded independently without reference to other pictures. MPEG encoding classifies pictures into three types: I, P, and B. I represents intra-coded frames, P represents forward-predicted frames, and B represents bidirectionally interpolated frames. An I-frame is a keyframe and can be considered a complete video frame. P-frames and B-frames record changes relative to an I-frame, representing the difference from the previous frame, while B-frames represent the difference between the previous and next frames. An I-frame serves as a reference for decoding subsequent P-frames and B-frames. Without an I-frame, P-frames and B-frames cannot be decoded. The length of a GOP is the distance between two consecutive I-frames. Assuming the video frame sequence is as follows: I, P, B, B, B, I, P, B, B, B, then the GOP length is 5, meaning that a GOP contains 5 video frames.

[0058] For ease of description, the operation of clearing the existing cache and calling the relevant API to control the decoder to perform decoding processing from the reference frame corresponding to the specified timeline position based on the specified timeline position indicated by the jump request is referred to as a seek operation.

[0059] In the traditional way, when the jump request misses the cache queue, the application will perform a seek operation. However, in some scenarios, if you want to achieve fast switching of video images, if you use the traditional method, the application will perform more unnecessary calculations, which takes a long time and causes serious video freezes and poor user experience. For example, when the jump request indicates a backward jump and the jump request misses the cache, the decoded video frames included in the cache queue belong to the same GOP as the video frames at the specified timeline position indicated by the jump request. If the traditional method is used, a seek operation will be performed. Using the traditional method, from the I frame along the timeline backward to the last video frame in the cache queue, the video frames in this range are repeatedly decoded, which consumes a lot of calculation time, and causes serious video freezes.

[0060] In summary, analyzing the rationality and necessity of executing seek operations and executing seek operations in the most necessary scenarios are extremely important for improving the processing efficiency of video screen switching and solving the problem of video screen freezes.

[0061] Based on this, the present disclosure provides a method and device for processing video jump display. The method uses a cache queue to perceive the scenarios where a seek operation actually needs to be performed, thereby improving the rationality and necessity of performing the seek operation, thereby improving the application's response speed to jump requests, reducing video screen freezes, and improving user experience.

[0062] Exemplarily, the video jump display processing method provided in the present disclosure is executed by an electronic device. The electronic device can be a tablet computer, a mobile phone (such as a foldable screen mobile phone, a large screen mobile phone, etc.), a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart TV, a smart screen, an HDTV, a 4K TV, a smart speaker, a smart projector, and other Internet of Things (IoT) devices. The present disclosure does not impose any restrictions on the specific type of electronic device.

[0063] Figure 1 is a schematic diagram of the framework of the application provided by the present disclosure; as shown in Figure 1, the application includes: a business layer, an intermediate layer and a decoding layer. The application can be, but is not limited to, a video playback application or a video editing application. In the present disclosure, the business layer is mainly used to obtain the jump request triggered on the timeline of the video preview and pass the jump request to the intermediate layer, obtain the decoded video frame returned by the intermediate layer in response to the jump request and render it for display; the intermediate layer is mainly used to determine whether a seek operation needs to be performed based on the specified timeline position indicated by the jump request, the timeline position of the decoded video frame in the cache queue, the jump direction and other information, and if it is determined to be necessary, call the relevant API to perform the seek operation; the decoding layer is mainly used to respond to the instructions of the intermediate layer to perform decoding processing on the video frames in the decoding queue.

[0064] In the following embodiments, an electronic device is taken as an example, and a video-related application is installed in the electronic device. The application adopts the framework shown in FIG1 as an example to introduce the method provided by the present disclosure in detail.

[0065] FIG2 is a flowchart of a method for processing video jump display provided by an embodiment of the present disclosure.

[0066] Referring to FIG. 2 , the method of this embodiment may include:

[0067] S101. Obtain a jump request triggered on the timeline of a video preview; the jump request is used to request a jump from a first timeline position to a second timeline position.

[0068] The application's business layer can display an interactive interface on the electronic device's display screen. The user triggers a jump request by manipulating the timeline displayed on the interactive interface, such as the video playback timeline, preview timeline, and other components, by touching or operating an external device connected to the electronic device (such as a mouse). The application's business layer determines information such as the second timeline position and jump mode based on the user's operation on the timeline, and generates a jump request based on the determined information. The jump mode will be described in detail later.

[0069] It should be noted that the present disclosure does not limit the way in which users trigger video jumps. In addition to the above examples, users can also trigger jump requests through the full-screen drag function, material drag function, thumbnail drag function, etc. provided by the application.

[0070] S102: Perform matching according to the second timeline position and each decoded video frame in the cache queue to obtain a matching result.

[0071] The cache queue is used to cache decoded video frames output by the decoder. For example, referring to FIG3 , a certain video frame is used as a reference video frame. The position of the reference video frame on the time axis is shown by the black vertical line in FIG3 . The cache queue may include a forward cache and a backward cache corresponding to the reference video frame. The forward cache includes a video frame whose time axis position is smaller than the time axis position of the reference video frame, and the backward cache includes a video frame whose time axis position is larger than the time axis position of the reference video frame. The forward cache and the backward cache may satisfy a preset proportional relationship. For example, the forward cache occupies 0.7 of the cache queue, and the backward cache occupies 0.3 of the cache queue.

[0072] In the present disclosure, the middle layer of the application program senses internally and determines whether a seek operation needs to be performed, breaking the traditional dependence on the business layer. The module in the middle layer that can quickly and accurately sense whether a seek operation needs to be performed is the cache queue. Therefore, when the middle layer of the application program receives a jump request, it queries the cache queue based on the second timeline position indicated by the jump request to obtain a matching result.

[0073] Among them, the purpose of querying the cache queue is to obtain two aspects of information: 1. Whether the jump request hits the cache queue, that is, whether the cache queue includes the video frame at the second timeline position; 2. Whether the video frame at the second timeline position and the decoded video frames in the cache queue belong to the same picture group.

[0074] The cache queue can determine whether the jump request hits the cache queue by comparing the second timeline position with the timeline position of the first video frame in the cache queue, and comparing the second timeline position with the timeline position of the last video frame in the cache queue. If it hits, it can simultaneously determine whether the video frame at the second timeline position and each decoded video frame in the cache queue belong to the same GOP; if it does not hit, the timeline position of the corresponding I frame and the time interval of the I frame are obtained, and it is determined whether the second timeline position is greater than the timeline position of the I frame and less than the timeline position of the next I frame to determine whether the second timeline position belongs to the GOP, and whether the cache queue belongs to the GOP, thereby obtaining the above-mentioned second aspect of information.

[0075] S103: If the first condition or the second condition is met, clear the cache queue, control the decoder to perform decoding based on the second timeline position, and store the decoded video frame in the cache queue. The first condition is that the jump request indicates a forward jump and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position does not belong to the same picture group as the cache queue.

[0076] S104: If the third condition or the fourth condition is met, the cache queue is not cleared. The third condition is that the cache queue includes the video frame at the second timeline position; the fourth condition is that the jump direction is backward, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position belongs to the same picture group as the cache queue.

[0077] The application's middle layer determines whether a seek operation is required based on the jump direction indicated by the jump request passed by the business layer and the matching result fed back by the cache queue. Based on the jump direction and matching result, the following scenarios can be classified:

[0078] Scenario 1: Jump forward, the jump request does not hit the cache queue, the video frame with the first timestamp and all decoded video frames in the cache queue belong to the same GOP;

[0079] Scenario 2: Forward jump, jump request misses the cache queue, the video frame with the first timestamp and the decoded video frames in the cache queue do not belong to the same GOP;

[0080] Scenario 3: Backward jump, jump request misses the cache queue, the video frame with the first timestamp and the decoded video frames in the cache queue do not belong to the same GOP;

[0081] Scenario 4: Backward jump, jump request misses the cache queue, the video frame with the first timestamp and all decoded video frames in the cache queue belong to the same GOP;

[0082] Scenario 5: Forward jump, jump request hits the cache queue;

[0083] Scenario 6: Backward jump, jump request hits the cache queue.

[0084] When the middle layer of the application determines that the current scene is any one of scenes 1 to 3, it determines that a seek operation needs to be performed; when the middle layer determines that the current scene is any one of scenes 4 to 6, it determines that a seek operation does not need to be performed.

[0085] After determining that a seek operation needs to be performed, the middle layer (cache queue) sends a jump request and an indication signal to the decoding layer. The indication signal indicates that the current cache queue is in a frame-missing state. The indication signal can also indicate a forward frame missing or a backward frame missing. The decoding layer determines the decoding start frame based on the second time axis and the frame-missing direction indicated by the jump request, starts decoding from the decoding start frame, and writes the decoded video frame output by the decoder into the cache queue. Among them, the decoding start frame is usually the first video frame (I frame) of the GOP, which is the reference frame for decoding.

[0086] S105 : Read the video frame at the second timeline position or any video frame within the deviation tolerance range corresponding to the second timeline position from the cache queue according to the satisfied condition and the jump mode corresponding to the jump request, and render and display the video frame.

[0087] The middle layer of the application can determine whether the video frame to be returned is a precise frame or an imprecise frame based on the conditions met by the jump request and the jump mode, and read the corresponding video frame from the cache queue and send it to the business layer, which will render and display it, thereby successfully realizing the screen jump display.

[0088] Among them, if the first condition or the second condition is met and the jump mode is the precise mode, the video frame at the second time axis position is read from the cache queue and rendered and displayed; if the first condition or the second condition is met and the jump mode is the imprecise mode, any video frame within the allowable deviation range corresponding to the second time axis position is read from the cache queue and rendered and displayed; if the third condition is met, the video frame at the second time axis position is directly read from the cache queue and rendered and displayed; if the fourth condition is met, the current decoding task is not interrupted, and the decoder continues to decode the video frames after the last video frame in the cache queue continuously backward along the time axis, and writes the decoded video frame into the cache queue, and then reads the video frame at the second time axis position from the cache queue and renders and displays.

[0089] The method of this embodiment, through the cache queue, senses the scenario where a seek operation actually needs to be performed, thereby improving the rationality and necessity of performing the seek operation, and then promptly responds to the jump request and returns the target video frame that meets the jump request for rendering and display, thereby reducing the video screen freeze phenomenon and improving the user experience.

[0090] Next, we will introduce in detail how to read video frames when the jump mode is non-precise mode.

[0091] In this disclosure, the application's middle layer can provide two seek modes for the business layer to choose from: 1. Precise seek mode; 2. Sliding seek mode, also known as non-precise seek mode. The business layer can determine which seek mode to use based on the business scenario. For example, if frame-by-frame seek or fixed-frame seek scenarios require precise seek mode. If seek is triggered by the preview axis, full-screen dragging, or material dragging functions, and if high-quality image smoothness is required, non-precise seek mode can be used.

[0092] After the business layer determines which seek mode to use based on the business scenario, it can add mode indication information in the jump request so that the middle layer can determine the seek mode, and then determine which frame on the timeline or which time range of video frames needs to be returned; it should be understood that if the seek mode is the precise seek mode, the video frame at the second timeline position needs to be returned, and the video frames at other timeline positions cannot meet the scene requirements; if the seek mode is the non-precise seek mode, the returned video frame can deviate from the second timeline position, giving priority to ensuring the smoothness of the video picture.

[0093] The video frames returned in the non-precise seek mode can be understood as non-precise frames. Since backward seek can return any frame after the currently displayed video frame, the smoothness of the backward seek can be guaranteed. However, in the forward seek scenario, whether the non-precise frame selection is appropriate seriously affects the seek performance. For example, as shown in Figure 4, assuming that four seeks are performed forward along the timeline, the video frame returned by the first seek is located between the timeline position of the first seek and the timeline position of the second seek, and the video frame returned by the second seek is located before the timeline position of the fourth seek. Since the strategy for selecting non-precise frames is to select the closest video frame as much as possible, in order to prevent the picture from jumping back and forth, neither the third nor the fourth seek can return a video frame. The video screen will always display the video frame returned by the second seek. Therefore, in the forward seek scenario, how to select non-precise frames is the key to implementing non-precise seek.

[0094] To improve the performance of inexact seeks in forward seek scenarios, the present disclosure predicts the third timeline position specified by the next jump request and determines the tolerance range for selecting inexact frames based on the second and third timeline positions. As a possible implementation, calculating the tolerance range can be achieved through the following steps:

[0095] Step a1: Calculate a first time difference based on the second timeline position and the third timeline position of the video frame displayed in response to the previous jump request. This can be expressed as: Δt = t1 - t0, where Δt represents the first time difference, t1 represents the second timeline position, and t0 represents the third timeline position. It should be understood that the jump direction indicated by the previous jump request was forward.

[0096] Step a2: Determine, along the jump direction indicated by the jump request, a time range on the time axis starting from the second time axis position and having a distance from the second time axis position that satisfies the first time difference as the allowable deviation range. This can be expressed as: t2 = Δt + t1, where t2 represents an endpoint of the allowable deviation range and can be denoted as the fourth time axis position. The time range between t2 and t1 on the time axis falls within the allowable deviation range.

[0097] For example, assuming that this jump request indicates a forward jump, the positions of t2 and t1 on the time axis can be shown in FIG5 , and any video frame between t2 and t1 can be an inaccurate frame.

[0098] By comparing the embodiments shown in Figures 4 and 5, it can be seen that when it is determined that a seek operation needs to be performed and the jump mode is determined to be a non-precise seek mode, the present disclosure can obtain the corresponding deviation tolerance range by predicting the next jump request. Taking non-precise frames within the deviation tolerance range can improve the seek hit rate and avoid the video screen jumping back and forth.

[0099] It should be noted that non-precise seeks can also be implemented by having the business layer issue a deviation tolerance. Alternatively, the user can pre-configure a first time difference, so that the middle layer can determine the deviation tolerance based on the pre-configured first time difference. Of course, other methods can also be used to determine the deviation tolerance, not limited to the above examples.

[0100] Taking Figure 5 as an example, in a forward jump scenario, the decoder is triggered to perform decoding processing from the I frame. Due to the decoding order, the decoded data of the video frame at position t2 is usually written into the cache queue first. The middle layer will also preferentially detect the video frame at position t2 in the cache queue. Since the video frame at position t2 meets the allowable deviation range, the video frame at position t2 will be returned as the video frame that meets the jump request. It should be understood that if multiple video frames within the allowable deviation range are detected in the cache queue, the video frame closest to the second timeline position on the timeline will be returned as the target video frame.

[0101] To further improve seek performance, the present disclosure optimizes this through a frame-dropping strategy. Traditionally, the decoder would simply discard non-reference frames preceding the timeline position indicated by the jump request. However, this direct discarding can lead to errors in calculating the scheduled playback duration of video frames in the decoding queue. The scheduled playback duration of a video frame is calculated based on the scheduled playback times of two adjacent frames in the decoding queue. If some non-reference frames are discarded, the calculated scheduled playback duration of the video frame will be incorrect.

[0102] By optimizing the frame drop strategy, as shown in Figure 6, the decoder discards non-reference frames between the I frame and the video frame corresponding to the prediction point (e.g., position t2 in the aforementioned embodiment) when performing decoding based on jump requests and indication signals. This reduces the number of video frames to be decoded, reduces the decoding computational effort, and improves seek performance.

[0103] After the application returns a video frame, it needs to update a time threshold. Decoded video frames below this time threshold are discarded, and decoded video frames greater than or equal to this time threshold are written to the cache queue to update the cache queue. Traditionally, this time threshold is calculated based on the number of previous and next frames in the cache queue and the scheduled playback time (PTS) of each video frame in the cache queue to update the cache queue. When this traditional method is used in forward-jumping scenarios, a large number of decoded video frames cannot enter the cache queue.

[0104] FIG7 is a flow chart of a method for processing video jump display provided by an embodiment of the present disclosure. Referring to FIG7 , the method of this embodiment includes:

[0105] S201: Calculate a target time threshold according to the maximum number of video frames that can be cached in the cache queue and the playback duration of a single video frame.

[0106] The length of the cache queue, the ratio of forward cache to backward cache, etc. are usually pre-set. The maximum number of video frames that can be cached forward is determined based on the ratio of the forward cache to the entire length of the cache queue and the length of the cache queue. The maximum number of video frames that can be cached forward is then multiplied by the predetermined single-frame playback duration to obtain the target time threshold.

[0107] For example, assuming that the forward buffer can buffer up to 20 frames, the playback duration of each frame is 33 milliseconds, and the target time threshold = 20*0.033 = 0.66 seconds.

[0108] S202: Calculate a second time difference according to the time axis position of the video frame output by the decoder and the time axis position of the video frame read in response to the jump request.

[0109] S203 : Compare the second time difference with the target time threshold to obtain a comparison result, and write the video frame output by the decoder into a cache queue or discard the video frame output by the decoder according to the comparison result.

[0110] Among them, if the comparison result indicates that the second time difference is less than the target time threshold, the decoded video frame output by the decoder is written into the cache queue; if the comparison result indicates that the second time difference is greater than or equal to the target time threshold, the decoded video frame output by the decoder is discarded.

[0111] This embodiment calculates the target time threshold by the number of video frames that can be cached in the forward cache, so as to update the cache queue so that there are enough decoded video frames in the forward cache, thereby ensuring that the forward cache can satisfy the next forward jump as much as possible.

[0112] In a specific embodiment, it is assumed that the application generates four jump requests based on the user's continuous operations, and the four jump requests are all forward jumps, and the four jumps return t a , t b , t c , t d For the video frames at the four time positions, each time the video frame is returned, the target time threshold is determined by the method of the embodiment shown in FIG8 , and the forward cache in the cache queue is updated.

[0113] Because decoding anomalies may occur during the decoding process of the original video data, and when the jump mode is the precise seek mode, it is determined that the precise frame may not be obtained after the seek operation is performed.

[0114] FIG9 is a flow chart of a method for processing video jump display provided by another embodiment of the present disclosure. Referring to FIG9 , the method of this embodiment includes:

[0115] S301: Obtain a jump request triggered on the timeline of a video preview; the jump request is used to request a jump from a first timeline position to a second timeline position.

[0116] S302: Perform matching according to the second timeline position and each decoded video frame in the cache queue to obtain a matching result.

[0117] S303: If the first condition or the second condition is met, clear the cache queue and control the decoder to perform decoding based on the second timeline position and store the decoded video frame into the cache queue.

[0118] Steps S301 to S303 of this embodiment are similar to steps S101 to S103 of the embodiment shown in FIG2 , and reference may be made to the detailed description of the embodiment shown in FIG2 . For the sake of brevity, they will not be repeated here.

[0119] S304: Determine whether the video frame at the second timeline position can be successfully obtained based on the number of decoded video frames, the maximum range of reference frames, the second timeline position, the scheduled playback time of the first video frame in the cache queue, and the scheduled playback duration of a single video frame.

[0120] When the jump mode is determined to be the precise mode according to the jump request, it is determined whether the precise frame can be successfully obtained. For example, it is determined whether the following conditions are currently met based on the above information:

[0121] Condition 1: The number of decoded video frames in the decoding queue is greater than the maximum range of reference frames;

[0122] Condition 2: The first timestamp is less than the scheduled playback time of the first video frame in the cache queue;

[0123] Condition 3: The time interval between the scheduled playback time of the first video frame in the cache queue and the first timestamp is less than the scheduled playback duration of a single video frame;

[0124] Condition 4: The scheduled playback time and the first timestamp of the first video frame in the cache queue are less than a preset time offset, and the preset time offset is equal to the product of the maximum range of the reference frame and the scheduled playback duration of a single video frame.

[0125] If conditions 1 to 4 are met simultaneously, it is determined that the video frame at the second timeline position cannot be successfully obtained for precise seek, and step S305 is executed. The inability to read the precise frame corresponding to the jump request may be due to an error in the maximum range of the reference frame provided by the video, which in turn causes a decoding error.

[0126] It should be noted that in precise seek mode, if any of the above conditions is not met, it means that the middle layer of the application can implement precise seek.

[0127] S305: Read the first video frame in the cache queue and render it for display.

[0128] The middle layer of the application reads the decoded data of the first video frame in the cache queue and sends it to the business layer, which renders and displays it, thus successfully achieving screen jump.

[0129] This embodiment is essentially a fault-tolerant processing mechanism for precise seek scenarios. In the forward jump scenario, when it is determined that the precise frame cannot be obtained, the video frame closest to the second timeline position in the cache queue is returned as the precise frame to ensure the smoothness of the video screen jump.

[0130] FIG10 is a structural diagram of a processing device for video jump display provided in an embodiment of the present disclosure. Referring to FIG10 , the device 1000 provided in this embodiment includes:

[0131] The acquisition module 1001 is used to acquire a jump request triggered on the time axis of the video preview; the jump request is used to request to jump from a first time axis position to a second time axis position;

[0132] A matching module 1002 is configured to perform matching based on the second timeline position and each decoded video frame in the cache queue to obtain a matching result;

[0133] Processing module 1003 is configured to clear the cache queue if a first condition or a second condition is met, and control the decoder to perform decoding based on the second timeline position and store the decoded video frame in the cache queue; and not clear the cache queue if a third condition or a fourth condition is met; the first condition is that the jump request indicates a forward jump, and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position and the cache queue do not belong to the same picture group; the third condition is that the cache queue includes the video frame at the second timeline position; the fourth condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position and the cache queue belong to the same picture group; and, based on the conditions met and the jump mode corresponding to the jump request, read the video frame at the second timeline position or any video frame within a deviation tolerance range corresponding to the second timeline position from the cache queue and render and display it;

[0134] The display module 1004 is configured to display corresponding video frames.

[0135] In some embodiments, the processing module 1003 is specifically used to: if the first condition or the second condition is met and the jump mode is the precise mode, then read the video frame at the second timeline position from the cache queue and render and display it; if the first condition or the second condition is met and the jump mode is the imprecise mode, then read any video frame within the allowable deviation range corresponding to the second timeline position from the cache queue and render and display it; if the third condition or the fourth condition is met, then read the video frame at the second timeline position from the cache queue and render and display it.

[0136] In some embodiments, before reading any video frame within the deviation tolerance range corresponding to the second timeline position from the cache queue and rendering and displaying it, the processing module 1003 is further used to calculate the deviation tolerance range corresponding to the second timeline position.

[0137] In some embodiments, the processing module 1003 is specifically used to: calculate a first time difference based on the second time axis position and the third time axis position, where the third time axis position is the time axis position of the video frame displayed by the previous jump request; along the jump direction, determine that the time range starting from the second time axis position and the time range on the time axis that is away from the second time axis position and satisfies the first time difference is the allowable deviation range.

[0138] In some embodiments, the processing module 1003 is specifically configured to: if the jump direction is forward jump, control the decoder to discard the non-reference frame located before the deviation tolerance on the time axis during decoding.

[0139] In some embodiments, the processing module 1003 is also used to: calculate a target time threshold based on the maximum number of video frames that can be cached forward by the cache queue and the playback duration of a single video frame; calculate a second time difference based on the time axis position of the video frame output by the decoder and the time axis position of the video frame read in response to the jump request; compare the second time difference with the target time threshold to obtain a comparison result, and write the video frame output by the decoder into the cache queue or discard the video frame output by the decoder according to the comparison result.

[0140] In some embodiments, the processing module 1003 is also used to: if it is determined that the video frame at the second timeline position cannot be successfully obtained based on the number of decoded video frames, the maximum range of reference frames, the second timeline position, the scheduled playback time of the first video frame in the cache queue, and the scheduled playback duration of a single video frame, then the first video frame in the cache queue is read and rendered for display.

[0141] The device provided in this embodiment can be used to implement the technical solution of any of the aforementioned method embodiments. Its implementation principles and technical effects are similar. Please refer to the detailed description of the aforementioned method embodiments. For the sake of brevity, they will not be repeated here.

[0142] Exemplarily, the present disclosure provides an electronic device comprising: one or more processors; a memory; and one or more computer programs; wherein the one or more computer programs are stored in the memory; when the one or more processors execute the one or more computer programs, the electronic device implements the processing method for video jump display of the previous embodiment.

[0143] Illustratively, the present disclosure provides a chip system, which is applied to an electronic device including a memory and a sensor; the chip system includes: a processor; and a processing method for video jump display when the processor executes the above-mentioned embodiment.

[0144] Exemplarily, the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor in an electronic device, the processing method for video jump display in the above embodiment is implemented.

[0145] Exemplarily, the present disclosure provides a computer program product, which, when executed on a computer, enables the computer to execute the processing method for video jump display of the foregoing embodiment.

[0146] In the above embodiments, all or part of the functions can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state drive (SSD)), etc.

[0147] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0148] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing video jump display, characterized in that: include: Get the jump request triggered on the timeline of the video preview; The jump request is used to request to jump from the first time axis position to the second time axis position; Matching is performed according to the second timeline position and each decoded video frame in the cache queue to obtain a matching result; If the first condition or the second condition is met, the cache queue is cleared, and the decoder is controlled to decode based on the second timeline position and store the decoded video frame into the cache queue; if the third condition or the fourth condition is met, the cache queue is not cleared; The first condition is that the jump request indicates a forward jump, and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position and the cache queue do not belong to the same picture group; the third condition is that the cache queue includes the video frame at the second timeline position; the fourth condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position and the cache queue belong to the same picture group; According to the satisfied conditions and the jump mode corresponding to the jump request, the video frame at the second timeline position or any video frame within the deviation tolerance range corresponding to the second timeline position is read from the cache queue and rendered for display.

2. The method according to claim 1, characterized in that The step of reading the video frame at the second timeline position or any video frame within the deviation tolerance range corresponding to the second timeline position from the cache queue and rendering and displaying the video frame according to the satisfied condition and the jump mode indicated by the jump request includes: If the first condition or the second condition is met, and the jump mode is the precise mode, reading the video frame at the second timeline position from the cache queue and rendering it for display; If the first condition or the second condition is met, and the jump mode is the non-precise mode formula, read any video frame within the deviation tolerance range corresponding to the second timeline position from the cache queue and render it for display; If the third condition or the fourth condition is met, the video frame at the second timeline position is read from the cache queue and rendered for display.

3. The method according to claim 2, characterized in that Before reading any video frame within the deviation tolerance range corresponding to the second timeline position from the cache queue and rendering and displaying it, the method further includes: The deviation tolerance range corresponding to the second time axis position is calculated.

4. The method according to claim 3, characterized in that The calculating the deviation tolerance range corresponding to the second time axis position includes: Calculating a first time difference according to the second time axis position and a third time axis position, wherein the third time axis position is the time axis position of the video frame displayed by the previous jump request; Along the jump direction, determine that the time range starting from the second time axis position and the time range whose distance from the second time axis position satisfies the first time difference is the deviation tolerance range.

5. The method according to claim 1, characterized in that The controlling the decoder to perform decoding based on the second time axis position includes: If the jump direction is a forward jump, the decoder is controlled to discard non-reference frames located before the deviation tolerance range on the time axis during decoding.

6. The method according to claim 1, characterized in that The method further comprises: Calculate the target time threshold according to the maximum number of video frames that can be cached forward by the cache queue and the playback duration of a single video frame; A second time difference is calculated based on the time axis position of the video frame output by the decoder and the time axis position of the video frame read in response to the jump request; The second time difference value is compared with the target time threshold to obtain a comparison result, and the video frame output by the decoder is written into the cache queue or the video frame output by the decoder is discarded according to the comparison result.

7. The method according to claim 2, characterized in that The method further comprises: If it is determined that the video frame at the second timeline position cannot be successfully obtained based on the number of decoded video frames, the maximum range of reference frames, the second timeline position, the scheduled playback time of the first video frame in the cache queue, and the scheduled playback duration of a single video frame, the first video frame in the cache queue is read and rendered for display.

8. A processing device for video jump display, characterized in that: include: An acquisition module, used to acquire a jump request triggered on the timeline of a video preview; The jump request is used to request to jump from the first time axis position to the second time axis position; A matching module, configured to perform matching according to the second timeline position and each decoded video frame in the cache queue to obtain a matching result; A processing module, configured to clear the cache queue if the first condition or the second condition is met, and control the decoder to decode based on the second timeline position and store the decoded video frame into the cache queue; and not clear the cache queue if the third condition or the fourth condition is met; The first condition is that the jump request indicates a forward jump, and the matching result indicates that the cache queue does not include the video frame at the second timeline position; the second condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position and the cache queue do not belong to the same picture group; the third condition is that the cache queue includes the video frame at the second timeline position; the fourth condition is that the jump direction is a backward jump, the matching result indicates that the cache queue does not include the video frame at the second timeline position, and the video frame at the second timeline position and the cache queue belong to the same picture group; And, according to the satisfied conditions and the jump mode corresponding to the jump request, the video frame at the second timeline position or any video frame within the deviation tolerance range corresponding to the second timeline position is read from the cache queue and rendered for display; The display module is used to display the corresponding video frame.

9. An electronic device, characterized in that: include: Memory and processor; The memory is configured to store computer program instructions; The processor is configured to execute the computer program instructions so that the electronic device implements the video jump display processing method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: include: Computer program instructions; At least one processor of the electronic device executes the computer program instructions, so that the electronic device implements the video jump display processing method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that The electronic device runs the computer program product so that the electronic device implements the video jump display processing method as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Network computer multimedia video playing method and system

    CN111757144A

  • Video file playing method and device in progress dragging process

    CN112929755A

  • Video jump playing method and device, terminal equipment and storage medium

    CN114666666A

  • Method for processing dragging watching video in wireless communication environment

    CN115278349A

  • Video seek method and system, video playing server and storage medium

    CN115967842A