Decoded data acquisition method and apparatus, device and storage medium

Abstract

Disclosed in the present disclosure are a decoded data acquisition method and apparatus, a device, and a storage medium, for solving the problem in the prior art of low overall efficiency of NLE caused by low decoded data acquisition efficiency. The method comprises: decoding a target image frame in a material track clip to generate decoded data of the target image frame; on the basis of a multilevel cache policy, storing the decoded data into a multilevel cache space, the multilevel cache policy comprising: if decoded data that has been cached in a first storage space does not reach the upper limit of the first storage space, caching in the first storage space the decoded data of the target image frame, and if the decoded data that has been cached in the first storage space reaches the upper limit of the first storage space, caching in a second storage space the decoded data of the target image frame; determining a target storage space from the multilevel cache space; and reading the decoded data of the target image frame from the target storage space, rendering the target image frame on the basis of the decoded data and displaying a preview image.

Description

Decoded data acquisition method, device, equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese invention patent application with application number 202310776768.7, entitled “Decoding data acquisition method, device, equipment and storage medium” and filing date June 28, 2023, and the entire application is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a method and device for obtaining decoding data, an electronic device, and a computer-readable storage medium. Background Art

[0004] When a user uses a non-linear editing (NLE) system to edit materials such as video files, the user may perform a seek operation on the materials loaded into a video editing track.

[0005] Performing a seek operation on a clip loaded onto a video editing track involves dragging the timeline on the video editing track, or dragging the clip loaded onto the track, to change the relative position of the timeline and the clip along the timeline of the video editing track. This operation then locates the clip at the target playback position based on the timeline position, and then renders and presents at least one image frame of the clip at the target playback position. The clip loaded onto the video editing track is referred to as a clip track clip.

[0006] According to the existing technology, after performing a seek operation, the NLE system will decode the material currently loaded into the memory in real time according to the located target playback position, so as to obtain the decoded data of the image frame corresponding to the target playback position, and then render and present the image frame based on the decoded data.

[0007] The method of real-time decoding of video files to obtain decoded data of image frames matching the target playback position has the problem of low efficiency in obtaining decoded data, thereby reducing the overall efficiency of NLE.

[0008] Summary of the Invention

[0009] The embodiments of the present disclosure provide a method for obtaining decoded data, which is used to solve the problem in the prior art that, in response to a preview operation triggered on a video editing track, real-time decoding of material is used to obtain decoded data of image frames that match a target playback position, resulting in low decoded data acquisition efficiency and, in turn, low overall NLE efficiency.

[0010] The embodiments of the present disclosure also provide a decoding data acquisition device, an electronic device, and a computer-readable storage medium.

[0011] The embodiments of the present disclosure adopt the following technical solutions:

[0012] A method for obtaining decoded data, comprising:

[0013] For a material track segment presented on a video editing track, decoding a target frame image in the material track segment to generate decoded data of the target frame image;

[0014] According to a multi-level cache strategy, the decoded data of the target frame image is stored in a multi-level cache space; wherein the multi-level cache space includes at least a first storage space and a second storage space, and the read and write speed of the first storage space is greater than the read and write speed of the second storage space; the multi-level cache strategy includes: if the decoded data of the frame image cached in the first storage space has not reached the upper limit of the first storage space, then the decoded data of the target frame image is cached in the first storage space; if the decoded data of the frame image cached in the first storage space has reached the upper limit of the first storage space, then the decoded data of the target frame image is cached in the second storage space;

[0015] In response to a preview operation triggered on a video editing track, if a timeline position of the target frame image in the material track segment is previewed, determining a target storage space from the multi-level cache space, the target storage space being used to store decoded data of the target frame image;

[0016] If the decoded data of the target frame image is stored in the multi-level cache space, the decoded data of the target frame image is read from the target storage space, and the target frame image is rendered based on the decoded data and a preview image is displayed.

[0017] A decoding data acquisition device, comprising:

[0018] a decoding unit, configured to decode a target frame image in a material track segment presented on a video editing track, and generate decoded data of the target frame image;

[0019] A storage unit, configured to store the decoded data of the target frame image in a multi-level cache space according to a multi-level cache strategy; wherein the multi-level cache space includes at least a first storage space and a second storage space, and the read and write speed of the first storage space is greater than the read and write speed of the second storage space; the multi-level cache strategy includes: if the decoded data of the frame image cached in the first storage space has not reached an upper limit of the first storage space, then caching the decoded data of the target frame image in the first storage space; if the decoded data of the frame image cached in the first storage space has reached the upper limit of the first storage space, then caching the decoded data of the target frame image in the second storage space;

[0020] a storage space determining unit, configured to, in response to a preview operation triggered on a video editing track, determine a target storage space from the multi-level cache space if the timeline position of the target frame image in the material track segment is previewed, the target storage space being used to store decoded data of the target frame image;

[0021] A data acquisition and display unit is used to read the decoded data of the target frame image from the target storage space if the decoded data of the target frame image is stored in the multi-level cache space, and render the target frame image based on the decoded data and display a preview image.

[0022] An electronic device, comprising: a memory and a processor, wherein:

[0023] The memory is used to store programs; the processor is coupled to the memory and is used to execute the programs stored in the memory to perform the above-mentioned decoding data acquisition method.

[0024] A computer-readable storage medium storing a computer program, wherein the computer program can implement the above-mentioned decoding data acquisition method when executed by a computer.

[0025] At least one of the above technical solutions adopted in the embodiments of the present disclosure can achieve the following beneficial effects:

[0026] Since a multi-level cache space including at least a first storage space and a second storage space is used to store the decoded data generated by decoding the target frame image in the material track segment, based on this, in response to the preview operation triggered on the video editing track, when the timeline position of the target frame image in the material track segment is previewed, if the decoded data of the target frame image is stored in the multi-level cache space, then the decoded data of the target frame image can be obtained from the target storage space of the multi-level cache space to render and preview the target frame, without the need to use real-time decoding of video files to obtain the decoded data of the target frame image as in the prior art, thereby improving the efficiency of obtaining the decoded data, and thus improving the overall efficiency of the NLE. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0028] FIG1a is a flowchart of a specific implementation of a method for obtaining decoded data provided by an embodiment of the present disclosure;

[0029] FIG1b is a schematic diagram of an interface of a non-linear editing system;

[0030] FIG2 is a schematic diagram of a specific implementation process of the decoding data acquisition method provided by an embodiment of the present disclosure in a non-linear editing system;

[0031] FIG3 is a schematic diagram of a specific implementation process of the decoding data acquisition method provided by an embodiment of the present disclosure in a non-linear editing system;

[0032] FIG4 is a schematic diagram of the specific structure of a decoding data acquisition device provided by an embodiment of the present disclosure;

[0033] FIG5 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0035] It is apparent to those skilled in the art that, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0036] The terms "first," "second," and the like in the specification and claims of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, and this is merely a way of distinguishing objects of the same attributes when describing the embodiments of the present disclosure. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, so that a process, method, system, product, or apparatus that includes a series of units is not necessarily limited to those units, but may include other units that are not explicitly listed or inherent to those processes, methods, products, or apparatus.

[0037] To address the problem in the prior art that, in response to a preview operation triggered on a video editing track, real-time decoding of material is used to obtain decoded data of image frames matching a target playback position, resulting in low decoded data acquisition efficiency and, in turn, low overall NLE efficiency, the presently disclosed embodiments provide a decoded data acquisition method.

[0038] The execution subject of the method may be, for example, a computing device such as a mobile phone, a personal computer, a wearable device, or software installed on the computing device, such as a video file player or decoder, etc. The embodiments of the present disclosure do not limit the computing device or software that is the execution subject.

[0039] In addition, different steps of the method may be implemented by different computing devices or software, and the embodiments of the present disclosure do not limit which computing device or software is used to implement which step.

[0040] For ease of description, the method provided by the embodiment of the present disclosure is described in detail below by taking the execution subject of the method as a non-linear editing system installed on a computing device as an example.

[0041] Please refer to FIG1a of the specification, which is a flowchart of a specific implementation of a method for obtaining decoded data provided by an embodiment of the present disclosure, including the following steps:

[0042] Step 11: For the material track segment presented on the video editing track, the non-linear editing system decodes the target frame image in the material track segment to generate decoded data of the target frame image;

[0043] For step 11, in an optional embodiment, the non-linear editing system may, for example, decode the target image frame contained in the material track segment in response to a user's preview instruction for the material track segment presented on the video editing track, thereby generating decoded data of the target frame image.

[0044] In step 11, the "target frame image" may not specifically refer to a certain frame image, but generally refers to an image in the material track segment that is decoded by the linear editing system.

[0045] Step 12: The non-linear editing system stores the decoded data of the target frame image into the multi-level cache space according to the multi-level cache strategy;

[0046] The multi-level cache space mentioned here at least includes: a first storage space and a second storage space.

[0047] In the embodiment of the present disclosure, the read and write speed of the first storage space is greater than the read and write speed of the second storage space.

[0048] In a specific example, the first storage space may be, for example, a dedicated storage space; the second storage space may be, for example, a shared storage space.

[0049] The dedicated storage space mentioned here refers to storage space specifically used to store decoded data obtained by decoding the target frame image in the material track segment, and is not used to store other data. In contrast, shared storage space refers to storage space that can be used to store decoded data obtained by decoding the target frame image in the material track segment, but can also be used to store other data.

[0050] In an optional embodiment, by setting an identifier for the storage space to indicate whether the storage space is a dedicated storage space or a shared storage space, it can be indicated whether the storage space is a dedicated storage space or a shared storage space. When storing data in a dedicated storage space (or a shared storage space), the identifier can be used to determine whether the storage space is a dedicated storage space or a shared storage space, and further, based on the pre-set storage rule that "the dedicated storage space is specifically used to store decoded data obtained by decoding the target frame image in the material track segment, and is not used to store other data; and the shared storage space can be used to store decoded data obtained by decoding the target frame image in the material track segment, and can also be used to store other data", it can be determined whether the data to be stored can be stored in the dedicated storage space (or shared storage space).

[0051] Alternatively, dedicated storage space refers to storage space used by the graphics processing unit (GPU) of a computing device and not shared with other applications; while shared storage space refers to storage space that can be used by all applications but is used preferentially by the GPU.

[0052] In a specific example, the first storage space may be, for example, a video memory; the second storage space may be, for example, a memory.

[0053] In an embodiment of the present disclosure, a multi-level caching strategy may include, but is not limited to: if the decoded data of the frame image cached in the first storage space has not reached the upper limit of the first storage space, then the decoded data of the target frame image is cached in the first storage space; if the decoded data of the frame image cached in the first storage space has reached the upper limit of the first storage space, then the decoded data of the target frame image is cached in the second storage space.

[0054] Regarding the specific implementation of step 12, in an optional embodiment, the decoded data generated by executing step 11 can be first stored in the local memory of the computing device on which the non-linear editing system is installed. Thus, the non-linear editing system can retrieve the decoded data from the memory and then store the decoded data in the multi-level cache space. Of course, when the second storage space is the internal memory, and the decoded data of the frame image stored in the first storage space has reached the upper limit of the first storage space, the non-linear editing system may not perform the operation of retrieving the decoded data from the internal memory and storing it in the second storage space.

[0055] In an optional embodiment, to achieve flexible collaborative storage of data, the first storage space may include: a first sub-storage space and a second sub-storage space. In a specific example, when the first storage space is video memory, the first sub-storage space is dedicated video memory, and the second sub-storage space is shared video memory.

[0056] Among them, dedicated video memory refers to the memory used by the GPU of the computing device itself and not shared with other applications; while shared video memory refers to the memory that can be used by all applications but is used by the GPU first.

[0057] In an optional embodiment, when the first storage space includes a first sub-storage space and a second sub-storage space, the multi-level caching strategy described in the embodiment of the present disclosure may also include: if the decoded data of the frame image cached in the first sub-storage space has reached the upper limit of the first sub-storage space, then caching the decoded data of the target frame image in the second sub-storage space.

[0058] When the first storage space includes a first sub-storage space and a second sub-storage space, the multi-level cache space in the disclosed embodiments is equivalent to being composed of at least three types of storage space: a first sub-storage space, a second sub-storage space, and a second storage space. In a specific example, the first sub-storage space, the second sub-storage space, and the second storage space can be dedicated video memory, shared video memory, and main memory, respectively.

[0059] In the embodiment of the present disclosure, before storing the decoded data in the multi-level cache space, the non-linear editing system may implement the following steps to designate the storage space as the multi-level cache space:

[0060] First, the nonlinear editing system obtains the size of the currently available volatile storage space (including memory and video memory) of the computing device. For example, the size of the currently available volatile storage space, that is, the size of the currently free volatile storage space, can be obtained from the computing device's global resource management center. The global resource management center can be a component of the computing device that has permission to obtain information such as volatile storage space usage.

[0061] Then, the nonlinear editing system can determine the multi-level cache space based on the size of the currently available volatile storage space (for example, 100M of memory and 50M of video memory) and the preset resource usage ratio (for example, 60%) - specifically 60% of the currently available 100M of memory, that is, 60M of memory; and 60% of the currently available 50M of video memory, that is, 30M of video memory.

[0062] After determining the multi-level cache space, the decoded data can be stored in the multi-level cache space for subsequent querying. It should be noted that the decoded data stored in the multi-level cache space can have an identifier of the decoded data, such as the frame number of the image frame corresponding to the decoded data; alternatively, the decoded data can be stored in the multi-level cache space in correspondence with the frame number of the corresponding image frame for subsequent querying.

[0063] It should be noted that, in the embodiment of the present disclosure, the purpose of storing the decoded data obtained by decoding the material track fragment in the multi-level cache space is mainly to take into account that if the non-linear editing system subsequently detects that a preview operation for the material track fragment is triggered on the video editing track, then when there is decoded data corresponding to the image to be previewed in the multi-level cache space, the decoded data can be directly obtained from the multi-level cache space without the need to decode the material track fragment in real time to obtain the decoded data, thereby ensuring the data acquisition efficiency of the video editing process.

[0064] The storage of decoded data in the disclosed embodiment has the following characteristics:

[0065] 1. The nonlinear editing system stores the decoded data obtained by decoding the images in the "material track clips presented on the video editing track", rather than the decoded data obtained by decoding the material that has not yet been loaded onto the video editing track;

[0066] 2. The decoded data stored in the multi-level cache space is the decoded data obtained by decoding during the video editing process, rather than the decoded data obtained by decoding in other processes, nor is it the decoded data obtained by decoding images that have not been previewed during the video editing process. Therefore, by adopting the decoded data storage method provided by the embodiment of the present disclosure, there is no need to deliberately decode the image in advance to obtain the decoded data, but the decoded data can be stored naturally along with the image preview process.

[0067] The above two features are both based on the consideration that in the "non-linear editing" scenario, users are likely to reposition and preview the material track segments at any time, for example, performing a seek operation as described in the background technology. In such a scenario and under such demand background, the decoded data that has been decoded before the seek operation is stored. When the seek operation occurs, the stored decoded data can be directly obtained for rendering and previewing the image, thereby improving the efficiency of non-linear editing.

[0068] Step 13: In response to the preview operation triggered on the video editing track, if the preview finds the timeline position of the target frame image in the material track segment, the nonlinear editing system determines the target storage space from the multi-level cache space;

[0069] Among them, the preview operation triggered on the video editing track may include but is not limited to the seek operation triggered on the video editing track, that is: by dragging the timeline on the video editing track, or by dragging the material loaded on the video editing track, so that the timeline position and the material change relative positions along the time axis direction of the video editing track, and then locating the target playback position of the material according to the timeline position, and then rendering and presenting at least the image frame in the material located at the target playback position.

[0070] Of course, in addition to operations such as dragging the timeline or dragging the material, other operations that can cause the timeline position and the material to change relative positions along the time axis direction of the video editing track, thereby re-positioning the target playback position of the material according to the timeline position, and then rendering and at least presenting the image frame in the material at the target playback position, can all be classified as the "preview operation" described in the embodiments of the present disclosure.

[0071] To help readers understand the relationship between preview operations triggered on a video editing track and timeline positions, please refer to Figure 1b of this specification. Figure 1b is a schematic diagram of a material track clip displayed on a video editing track in a nonlinear editing system. As shown, the image frames contained in the material track clip can be previewed in the preview window. Below this playback window are control buttons that support playback control of the video file.

[0072] In actual scenarios, the user may directly drag the timeline as shown in Figure 1b, and the dragging operation is the preview operation described in step 13; alternatively, the user may also long press the "rewind play" button in the control button, and the long press operation of the button is also the preview operation described in step 13.

[0073] In actual scenarios, the user can perform the preview operation in any manner, and the embodiments of the present disclosure do not limit how to perform the preview operation.

[0074] As previously explained, the target frame image mentioned in step 13 does not specifically refer to a specific frame image, but rather refers generally to the image decoded by the linear editing system within the material track segment. In step 13, if the preview operation causes the timeline position to coincide with the location of an image within the material track segment, the linear editing system will generally sequentially retrieve the decoded data for that image, the decoded data for other images that belong to the same material track segment but follow that image, and the decoded data for images located on other video editing tracks and that follow the material track segment to which the image belongs. The system then renders preview images for each image based on the retrieved decoded data.

[0075] If, triggered by the preview operation, the decoded data to be acquired by the linear editing system includes the decoded data of the target image described in step 1, and the decoded data of the target image exists in the multi-level cache space, then the linear editing system may determine the target storage space from the multi-level cache space and then execute the subsequent step 14. If, triggered by the preview operation, the decoded data to be acquired by the linear editing system does not include the decoded data of the target image described in step 1, or, although it includes the decoded data of the target image described in step 1, the decoded data does not yet exist in the multi-level cache space, then the linear editing system no longer executes the subsequent step 14, but instead acquires the decoded data according to a conventional decoded data acquisition method—for example, acquiring an image frame from an image frame currently loaded into the memory for decoding to obtain the decoded data.

[0076] In the embodiment of the present disclosure, whether the decoded data of the target image exists in the multi-level cache space can be determined by the linear editing system by querying the multi-level cache space according to the identifier of the target image.

[0077] In the embodiment of the present disclosure, the identifier of the target image may refer to the number of the target image in the material track segment to which the target image belongs, for example, it may be the frame number of the image frame.

[0078] In the embodiment of the present disclosure, assuming that the decoded data of the target frame image has been stored in the multi-level cache space by executing step 12 and has not yet been consumed, then either of the two situations can trigger the non-linear editing system to determine the target storage space storing the target frame image from the multi-level cache space: 1. The timeline position just overlaps with the position of the target frame image; 2. The timeline position overlaps with the position of the image arranged before the target frame image in the material track segment.

[0079] The target storage space mentioned above refers to a storage space used to store decoded data of a target frame image, which may be a first storage space or a second storage space.

[0080] As previously mentioned, in the disclosed embodiments, a multi-level cache strategy is employed when storing decoded data. According to this multi-level cache strategy, decoded data is preferentially stored in the first storage space with a relatively high read / write speed until the first storage space reaches its storage limit, at which point the decoded data is stored in the second storage space.

[0081] It is understood that, from the perspective of ensuring decoded data acquisition efficiency, if decoded data can always be read from the first storage space with a faster read / write speed, the decoded data acquisition efficiency can be maximized. Therefore, in the embodiment of the present disclosure, if the decoded data stored in the first storage space is consumed, the decoded data stored in the second storage space is transferred to the free storage locations in the first storage space that have been released according to the first-in-first-out (FIFO) rule.

[0082] This data transfer method is adopted to take into account the process of previewing the image frames of the material track fragments on the video editing track, which conforms to the feature of "previewing sequentially starting from the timeline position". Based on this feature, if the decoded data of the previewed image frames can be moved from the second storage space to the first storage space in sequence in a FIFO manner, then in theory, the decoded data can always be read (i.e., consumed) from the first storage space with a faster reading and writing speed, thereby ensuring the efficiency of decoding data acquisition and thus ensuring the overall efficiency of NLE.

[0083] Corresponding to the above-mentioned sequential transfer method of decoded data, an optional implementation method of determining the target storage space from the multi-level cache space in step 13 may include the following sub-steps 1 and 2:

[0084] Sub-step 1: query the decoded data of the target frame image from the target storage space according to the data query strategy;

[0085] Sub-step 2: Determine the storage space where the decoded data of the queried target frame image is located as the target storage space.

[0086] The data query strategy may include: querying different storage spaces included in the multi-level cache space in order of the read and write speed of the storage space from large to small.

[0087] According to the data query strategy, if the decoded data of the target frame image is found in the multi-level cache space, then the specific storage space where the decoded data is located can be determined, that is, the target storage space is determined, and then step 14 is executed.

[0088] Step 14: If the decoded data of the target frame image is stored in the multi-level cache space, the nonlinear editing system reads the decoded data of the target frame image from the target storage space, and renders the target frame image based on the decoded data of the target frame image and displays a preview image.

[0089] In an optional embodiment, when performing step 12 to store the decoded data of the target frame image, the identifier of the target frame image and the decoded data may be stored in correspondence. Thus, in step 14, based on the identifier of the target frame image, the decoded data stored in correspondence with the identifier may be read from the target storage space as the decoded data of the target frame image.

[0090] After the decoded data of the target frame image is read, the target frame image can be rendered based on the decoded data and a preview image can be displayed.

[0091] In an optional embodiment, the nonlinear editing system can read the decoded data in the target storage space into the graphics processor memory (referred to as video memory), so that the graphics processor can render a preview image corresponding to the target frame image based on the decoded data.

[0092] Considering that the decoded data stored in the multi-level cache space may contain decoded data that has not been acquired and used (acquisition and use can be called consumption) by the non-linear editing system from the multi-level cache space in the recent period, such decoded data is considered "cold data". Cold data continues to occupy storage resources, which is equivalent to causing a waste of storage resources. Therefore, in an optional embodiment, the non-linear editing system can clear the decoded data that meets the data elimination rules from the multi-level cache space according to the data elimination rules set based on the Least Recently Used (LRU) algorithm.

[0093] The LRU algorithm refers to replacing the least used data. In the embodiment of the present disclosure, the data elimination rule set based on the LRU algorithm may include: eliminating the least used data and replacing the least used data with the most recently decoded data.

[0094] By adopting the above elimination method, it is possible to ensure that the storage resources of the multi-level cache space are fully utilized, and avoid the problem of infrequently used "cold data" occupying storage resources and causing resource waste.

[0095] In an optional embodiment, when a nonlinear editing system acquires decoded data for rendering, it is specifically triggered by the nonlinear editing system to achieve this. For the graphics processor, acquiring decoded data from video memory for rendering is more efficient than acquiring decoded data from memory. This is because if the graphics processor acquires decoded data from the memory of the computing device, it reads the acquired decoded data from the memory into the video memory, and then acquires the decoded data from the video memory for rendering. Therefore, from the perspective of the graphics processor, when both the memory and the video memory can be used to store decoded data, prioritizing the storage of the decoded data in the video memory is more conducive to the graphics processor efficiently acquiring the decoded data.

[0096] Based on the above considerations, in the disclosed embodiments, the video memory is defined as a storage space with a high consumption priority within the multi-level cache space, while the main memory is defined as a secondary storage space with a low consumption priority within the multi-level cache space. The "consumption priority" mentioned here can be equivalent to the "read and write speed" mentioned above, specifically referring to the read and write speed of data read and written by the graphics processor.

[0097] Furthermore, if the first storage space includes dedicated video memory (i.e., the first sub-storage space) and shared video memory (i.e., the second sub-storage space), the dedicated video memory can be further determined as the storage space with the highest consumption priority in the multi-level cache space, and the shared video memory can be determined as the storage space with the second highest consumption priority in the multi-level cache space.

[0098] It's important to note that the consumption priority of a multi-level cache is equivalent to the query priority followed by the nonlinear editing system when querying the decoded data stored in the multi-level cache. The higher the consumption priority, the higher the priority it receives when the nonlinear editing system queries for decoded data.

[0099] Based on the different types of storage spaces contained in the multi-level cache space having different consumption priorities, in an optional implementation, the non-linear editing system can query the decoding data of the target frame image from the target storage space according to the data query strategy; and determine the storage space where the decoding data of the queried target frame image is located as the target storage space.

[0100] The data query strategy includes querying different storage spaces included in the multi-level cache space in order of the read and write speed (ie, consumption priority) of the storage space from large to small.

[0101] With the above-mentioned method provided by the embodiment of the present disclosure, a multi-level cache space including at least a first storage space and a second storage space is used to store the decoded data generated by decoding the target frame image in the material track segment. Based on this, in response to the preview operation triggered on the video editing track, when the timeline position of the target frame image in the material track segment is previewed, if the decoded data of the target frame image is stored in the multi-level cache space, then the decoded data of the target frame image can be obtained from the target storage space of the multi-level cache space to render and preview the target frame, without the need to obtain the decoded data of the target frame image by real-time decoding of the video file as in the prior art, thereby improving the efficiency of obtaining the decoded data, and further improving the overall efficiency of the NLE.

[0102] To help readers better understand the method provided by the embodiments of the present disclosure, the specific implementation process of the above-mentioned method for obtaining decoded data in a non-linear editing system is introduced below in combination with actual scenarios.

[0103] Please refer to FIG2 for a schematic diagram of a specific implementation process of the decoding data acquisition method in practice. The execution subject of each step in the process may be, for example, a non-linear editing system. The process mainly includes the following steps:

[0104] Step 31: The non-linear editing system obtains the currently available dedicated video memory size, shared video memory size, and internal memory size from the global resource management center of the local device (such as the user's personal computer) where the non-linear editing system is installed.

[0105] Step 32: The nonlinear editing system sets / designates the dedicated video memory, shared video memory, and internal memory as a decoding buffer pool, and sets 60% of the currently available storage space of the dedicated video memory, shared video memory, and internal memory as the storage space size of the decoding buffer pool according to the default resource usage ratio of 60%.

[0106] In the embodiment of the present disclosure, the consumption priority of dedicated video memory, shared video memory, and internal memory may be in the order of: dedicated video memory > shared video memory > internal memory.

[0107] Step 33: The non-linear editing system stores decoded data of the video frames obtained by the decoder in the process of non-linear editing of the video file into a decoding buffer pool;

[0108] Among them, when storing the decoded data of the video frame in the decoding cache pool, the decoded data of the video frame can be stored in the decoding cache pool in the order of consumption priority from high to low, and in accordance with the rule of filling up the different types of storage space (including dedicated video memory, shared video memory, and internal memory) contained in the decoding cache pool in sequence.

[0109] The term "full storage" mentioned here refers to filling up 60% of the currently available storage space of the dedicated video memory, using the above example.

[0110] Step 34: The non-linear editing system receives a fast-rewind instruction during the non-linear editing of the video file;

[0111] Step 35: The non-linear editing system responds to the instruction and queries the decoding buffer pool according to the frame number of the video frame to be decoded corresponding to the instruction;

[0112] If it is found that the decoded data of the video frame with the frame number exists in the decoding cache pool, the decoded data of the video frame with the frame number is directly obtained from the decoding cache pool, and then based on the decoded data of the video frame, the corresponding preview image is rendered and presented; wherein, the decoded data of the video frame obtained from the decoding cache pool by the non-linear editing system will be deleted from the decoding cache pool as the decoded data of the consumed video frame, and then, the decoded data of the video frame stored in the decoding cache pool will be transferred in a FIFO manner - for details, please see the relevant description in the previous embodiment, which will not be repeated here.

[0113] If no decoded data of the video frame with the frame number is found in the decoding buffer pool, step 36 is executed;

[0114] Step 36: The non-linear editing system decodes the video file loaded in the memory of the computing device to obtain decoded data of the video frame with the frame number, and then performs an operation of rendering and presenting a corresponding image based on the video frame; in addition, the process jumps to step 37;

[0115] Step 37: The non-linear editing system stores the decoded data of the video frame with the frame number obtained by performing the decoding in step 36 into the decoding buffer pool.

[0116] It should be noted that the non-linear editing system can run a thread such as "Recyle Thread", which is used to monitor the usage of the decoded data of the video frames stored in the decoding cache pool; if the thread monitors that the decoded data of a certain video frame / frames stored in the decoding cache pool has been consumed the least number of times from the storage time to the current time compared with the decoded data of other video frames - for example, less than the preset threshold Cache Threshold, then the non-linear editing system can clear the decoded data of the video frames that meet the data elimination rules from the decoding cache pool according to the data elimination rules determined based on the LRU algorithm, that is, for example, eliminate the decoded data of the video frames that have been consumed the number of times less than the preset threshold Cache Threshold.

[0117] By adopting the above-mentioned scheme provided by the embodiment of the present disclosure, a decoding buffer pool is adopted to realize the storage of the decoded data of the video frames decoded in the process of decoding and presenting the video file by the non-linear editing system. Based on this, when an instruction is received during the process to trigger decoding of the video file to obtain the decoded data of the specified video frame, the decoded data of the specified video frame can be obtained from the decoding buffer pool, without having to adopt the real-time decoding of the video file to obtain the decoded data of the specified video frame as in the prior art, thereby improving the efficiency of obtaining the decoded data of the specified video frame in the process.

[0118] The specific implementation process of the above solution can also be further referred to the schematic diagram shown in FIG3 .

[0119] Based on the same inventive concept as the previous embodiments, in order to solve the problem in the prior art that, in response to a preview operation triggered on a video editing track, real-time decoding of materials is used to obtain decoded data of image frames matching the target playback position, resulting in low decoding data acquisition efficiency and thus low overall efficiency of the NLE, the embodiments of the present disclosure also provide a decoding data acquisition device.

[0120] The specific structural diagram of the device is shown in Figure 4, which includes the following functional units:

[0121] The decoding unit 41 is configured to decode a target frame image in a material track segment presented on a video editing track to generate decoded data of the target frame image;

[0122] The storage unit 42 is configured to store the decoded data of the target frame image in a multi-level cache space according to a multi-level cache strategy; wherein the multi-level cache space includes at least a first storage space and a second storage space, and the read and write speed of the first storage space is greater than the read and write speed of the second storage space; the multi-level cache strategy includes: if the decoded data of the frame image cached in the first storage space has not reached the upper limit of the first storage space, then caching the decoded data of the target frame image in the first storage space; if the decoded data of the frame image cached in the first storage space has reached the upper limit of the first storage space, then caching the decoded data of the target frame image in the second storage space;

[0123] a storage space determining unit 43 configured to, in response to a preview operation triggered on a video editing track, determine a target storage space from the multi-level cache space if the timeline position of the target frame image in the material track segment is previewed, the target storage space being used to store decoded data of the target frame image;

[0124] The data acquisition and display unit 44 is used to read the decoded data of the target frame image from the target storage space if the decoded data of the target frame image is stored in the multi-level cache space, and render the target frame image based on the decoded data and display a preview image.

[0125] In an optional implementation, the storage space determining unit 43 may be configured to:

[0126] According to the data query strategy, query the decoded data of the target frame image from the target storage space;

[0127] The storage space where the decoded data of the target frame image is found is determined as the target storage space.

[0128] The data query strategy includes querying different storage spaces included in the multi-level cache space in sequence according to the order of the read and write speed of the storage space from large to small.

[0129] In an optional implementation, the device provided by the embodiment of the present disclosure may further include:

[0130] The data transfer unit is used to transfer the decoded data stored in the second storage space to the free storage locations released in the first storage space in accordance with the first-in-first-out (FIFO) rule if the decoded data stored in the first storage space is consumed.

[0131] In an optional implementation, the first storage space is a dedicated storage space; the second storage space is a shared storage space; or, the first storage space is a video memory; and the second storage space is a memory.

[0132] In an optional embodiment, the first storage space includes: a first sub-storage space and a second sub-storage space;

[0133] When the first storage space is a video memory, the first sub-storage space is a dedicated video memory, and the second sub-storage space is a shared video memory;

[0134] The multi-level cache strategy further includes: if the decoded data of the frame image cached in the first sub-storage space has reached the upper limit of the first sub-storage space, then caching the decoded data of the target frame image in the second sub-storage space.

[0135] In an optional implementation, the device provided by the embodiment of the present disclosure may further include:

[0136] The data clearing unit is used to clear the decoded data that meets the data elimination rule from the multi-level cache space according to the data elimination rule set based on the least recently used (LRU) algorithm.

[0137] The above-mentioned decoded data acquisition device provided by the embodiment of the present disclosure adopts a multi-level cache space including at least a first storage space and a second storage space to store the decoded data generated by decoding the target frame image in the material track segment. Based on this, in response to the preview operation triggered on the video editing track, when the timeline position of the target frame image in the material track segment is previewed, if the decoded data of the target frame image is stored in the multi-level cache space, then the decoded data of the target frame image can be obtained from the target storage space of the multi-level cache space to render and preview the target frame, without the need to obtain the decoded data of the target frame image by real-time decoding of the video file as in the prior art, thereby improving the efficiency of obtaining the decoded data, and further improving the overall efficiency of the NLE.

[0138] Based on the same inventive concept as the previous embodiments, the embodiments of the present disclosure also provide an electronic device to solve the problem in the prior art that, in response to a preview operation triggered on a video editing track, real-time decoding of materials is used to obtain decoded data of image frames matching the target playback position, resulting in low efficiency in obtaining decoded data and thus low overall efficiency of the NLE.

[0139] As shown in FIG5 , the electronic device includes a memory 51 and a processor 52. The memory 51 can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0140] The processor 52 is coupled to the memory 51 and is used to execute the program stored in the memory 51 to execute the decoding data acquisition method described in the embodiment of the present disclosure, or execute the multimedia data storage method described in the embodiment of the present disclosure.

[0141] When the processor 52 executes the program in the memory 51 , in addition to the above functions, it can also implement other functions, for details, please refer to the description of the previous embodiments.

[0142] 5 , the electronic device further includes other components such as a display 54, a communication component 53, a power supply component 55, and an audio component 56. FIG5 only schematically illustrates some components, which does not mean that the electronic device only includes the components shown in FIG5 .

[0143] Accordingly, an embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the methods provided in the above embodiments.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0145] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

1. A method for obtaining decoded data, comprising: For a material track segment presented on a video editing track, decoding a target frame image in the material track segment to generate decoded data of the target frame image; According to the multi-level cache strategy, the decoded data of the target frame image is stored in the multi-level cache space; wherein the multi-level cache space at least includes a first storage space and a second storage space, and the read and write speed of the first storage space is greater than the read and write speed of the second storage space; the multi-level cache strategy includes: if the decoded data of the frame image cached in the first storage space does not reach the upper limit of the first storage space, then the decoded data of the target frame image is cached in the first storage space; if the decoded data of the frame image cached in the first storage space has reached the upper limit of the first storage space, then the decoded data of the target frame image is cached in the second storage space; In response to a preview operation triggered on a video editing track, if the timeline position of the target frame image in the material track segment is previewed, a target storage space is determined from the multi-level cache space, and the target storage space is used to store decoded data of the target frame image; If the decoded data of the target frame image is stored in the multi-level cache space, the decoded data of the target frame image is read from the target storage space, and the target frame image is rendered based on the decoded data and a preview image is displayed.

2. The method of claim 1, wherein: Determining a target storage space from the multi-level cache space includes: According to the data query strategy, query the decoded data of the target frame image from the target storage space; Determine the storage space where the decoded data of the target frame image is located as the target storage space; The data query strategy includes: querying different storage spaces included in the multi-level cache space in sequence according to the order of the read and write speed of the storage space from large to small.

3. The method of claim 2, further comprising: If the decoded data stored in the first storage space is consumed, the decoded data stored in the second storage space is sequentially transferred to the free storage positions released in the first storage space according to the first-in-first-out (FIFO) rule.

4. The method of claim 1, wherein: The first storage space is a dedicated storage space; the second storage space is a shared storage space; or, The first storage space is video memory; the second storage space is internal memory.

5. The method according to claim 1 or 4, wherein: The first storage space includes: a first sub-storage space and a second sub-storage space; When the first storage space is a video memory, the first sub-storage space is a dedicated video memory, and the second sub-storage space is a shared video memory; The multi-level cache strategy further includes: if the decoded data of the frame image cached in the first sub-storage space has reached the upper limit of the first sub-storage space, then caching the decoded data of the target frame image in the second sub-storage space.

6. The method of claim 1, further comprising: According to the data elimination rule set based on the least recently used (LRU) algorithm, the decoded data satisfying the data elimination rule is cleared from the multi-level cache space.

7. A decoding data acquisition device, wherein: include: A decoding unit, configured to decode a target frame image in a material track segment presented on a video editing track, and generate decoded data of the target frame image; A storage unit is used to store the decoded data of the target frame image in a multi-level cache space according to a multi-level cache strategy; wherein the multi-level cache space at least includes a first storage space and a second storage space The read and write speed of the first storage space is greater than the read and write speed of the second storage space; the multi-level cache strategy includes: if the decoded data of the frame image cached in the first storage space does not reach the upper limit of the first storage space, the decoded data of the target frame image is cached in the first storage space; if the decoded data of the frame image cached in the first storage space has reached the upper limit of the first storage space, the decoded data of the target frame image is cached in the second storage space; a storage space determination unit, configured to respond to a preview operation triggered on a video editing track, and if the timeline position of the target frame image in the material track segment is previewed, determine a target storage space from the multi-level cache space, wherein the target storage space is used to store decoded data of the target frame image; A data acquisition and display unit is used to read the decoded data of the target frame image from the target storage space if the decoded data of the target frame image is stored in the multi-level cache space, and render the target frame image based on the decoded data and display a preview image.

8. The device according to claim 7, wherein: The storage space determining unit is used to: According to the data query strategy, query the decoded data of the target frame image from the target storage space; Determine the storage space where the decoded data of the target frame image is located as the target storage space; The data query strategy includes: querying different storage spaces included in the multi-level cache space in sequence according to the order of the read and write speed of the storage space from large to small.

9. The apparatus of claim 8, further comprising: The data transfer unit is used to transfer the decoded data stored in the second storage space to the free storage positions released in the first storage space in sequence according to the first-in-first-out (FIFO) rule if the decoded data stored in the first storage space is consumed.

10. The device according to claim 7, wherein: The first storage space is a dedicated storage space; the second storage space is a shared storage space; or, The first storage space is video memory; the second storage space is internal memory.

11. The device according to claim 7 or 10, wherein: The first storage space includes: a first sub-storage space and a second sub-storage space; When the first storage space is a video memory, the first sub-storage space is a dedicated video memory, and the second sub-storage space is a shared video memory; The multi-level cache strategy further includes: if the decoded data of the frame image cached in the first sub-storage space has reached the upper limit of the first sub-storage space, then caching the decoded data of the target frame image in the second sub-storage space.

12. The apparatus of claim 7, further comprising: The data clearing unit is used to clear the decoded data that meets the data elimination rule from the multi-level cache space according to the data elimination rule set based on the least recently used LRU algorithm.

13. An electronic device, wherein: include: Memory and processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to execute the decoding data acquisition method described in any one of claims 1 to 6.

14. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer, can implement the method for obtaining decoded data as claimed in any one of claims 1 to 6.

Images