Frame memory management method and device, electronic equipment and computer program product

Abstract

The invention provides a frame memory management method and device, electronic equipment and a computer program product, and relates to the technical field of video encoding and decoding. The method comprises the following steps: determining the number of to-be-consumed recovery frames sent by a codec, wherein the number of the to-be-consumed recovery frames is determined based on the total frame number of a frame memory pool and the number of queue frames corresponding to a reference frame queue; obtaining a pre-configured threshold value of the number of sent frames; and dynamically adjusting the frame output operation of the codec according to the comparison result of the number of the to-be-consumed recovered frames and the threshold value of the number of the sent frames. According to the invention, the number of the to-be-consumed recovery frames sent by the codec is controlled, so that the problems of memory accumulation and overflow caused by the fact that a subsequent frame processing module cannot process the frames in time can be effectively avoided; in a resource-limited environment, stable operation of the system can be ensured, and collapse caused by memory exhaustion is avoided; and the resource utilization can be effectively optimized, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video coding, and particularly relates to an intra-frame memory management method, an intra-frame memory management device, an electronic device and a computer program product. BACKGROUND

[0002] In multimedia applications, the decoding module and the rendering module usually exist as two decoupled functional units. The decoding module is responsible for decoding compressed media data (such as a video stream) into raw frame data, while the rendering module is responsible for displaying these decoded frame data on the screen. This decoupled design improves the flexibility and maintainability of the system, but at the same time brings challenges in inter-module communication and resource management.

[0003] Under this architecture, after the decoding module processes a frame of data, it outputs the frame to the rendering module for consumption. Ideally, the rendering module should process these frames in a timely manner and release the related memory resources. After rendering is completed, the frame memory should be returned to the memory pool of the decoding module for reuse. SUMMARY

[0004] The present disclosure provides an intra-frame memory management method, device, electronic device, computer-readable storage medium and computer program product to at least solve the problems of memory overflow risk, low resource utilization efficiency, lack of dynamic adaptation capability and decreased user experience in the frame memory management scheme between the decoding module and the rendering module in the related art. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, an intra-frame memory management method is provided, comprising: determining the number of frames to be consumed and recycled sent by a codec, the number of frames to be consumed and recycled being determined based on the total number of frames of a frame memory pool and the number of frames corresponding to a reference frame queue; obtaining a pre-configured threshold value of the number of sent frames; and dynamically adjusting the frame output operation of the codec according to the comparison result of the number of frames to be consumed and recycled and the threshold value of the number of sent frames.

[0006] In an exemplary embodiment of the present disclosure, the determination of the number of frames to be consumed and recycled sent by the codec comprises: creating a frame memory pool of the codec and configuring the total number of frames corresponding to the frame memory pool; taking the number of frames required by a reference frame queue in the codec as the number of queue frames; and determining the number of frames to be consumed and recycled according to the difference between the total number of frames of the frame memory pool and the number of queue frames.

[0007] In an exemplary embodiment of the present disclosure, the obtaining of the pre-configured threshold value of the number of sent frames comprises: obtaining a device performance parameter of a media playback device and a content characteristic parameter corresponding to media content; and dynamically adjusting the threshold value of the number of sent frames according to the device performance parameter and the content characteristic parameter.

[0008] In an example embodiment of the present disclosure, the step of dynamically adjusting the frame output operation of the codec according to the comparison result of the number of frames to be consumed and recovered and the threshold of the number of frames sent comprises: when the number of frames to be consumed and recovered is greater than or equal to the threshold of the number of frames sent, suspending the frame output operation of the codec to the rendering module.

[0009] In an example embodiment of the present disclosure, the step of suspending the frame output operation of the codec to the rendering module comprises one or a combination of the following operations: blocking the codec thread corresponding to the codec; reducing the thread priority corresponding to the codec thread; and reducing the codec rate corresponding to the codec.

[0010] In an example embodiment of the present disclosure, the step of dynamically adjusting the frame output operation of the codec according to the comparison result of the number of frames to be consumed and recovered and the threshold of the number of frames sent further comprises: when the number of frames to be consumed and recovered is less than the threshold of the number of frames sent, resuming the frame output operation of the codec to the rendering module.

[0011] In an example embodiment of the present disclosure, the method further comprises: determining the frame life cycle state of each frame in each frame processing operation based on the frame state tracking mechanism, the frame processing operation comprising one or more of frame decoding operation, rendering consumption operation and frame recovery operation; and updating the frame state identifier according to the frame life cycle state, the frame life cycle state comprising frame processing state, frame consumption state and frame recovery state.

[0012] In an example embodiment of the present disclosure, the method further comprises: monitoring the module running state of the rendering module; when the module running state is in an abnormal state, determining the frame to be recovered based on the frame life cycle state and the state duration; and performing the frame recovery operation on the frame to be recovered.

[0013] According to a second aspect of the embodiments of the present disclosure, a frame memory management apparatus is provided, comprising: a frame number determination module configured to determine the number of frames to be consumed and recovered sent by a codec, the number of frames to be consumed and recovered being determined based on the total number of frames in a frame memory pool and the number of queue frames corresponding to a reference frame queue; a frame threshold acquisition module configured to acquire a pre-configured threshold of the number of frames sent; and a frame output control module configured to dynamically adjust the frame output operation of the codec according to the comparison result of the number of frames to be consumed and recovered and the threshold of the number of frames sent.

[0014] In an example embodiment of the present disclosure, the frame number determination module comprises a frame number determination unit configured to: create a frame memory pool of the codec, and configure a total frame number corresponding to the frame memory pool; take a frame number required by a reference frame queue in the codec as the queue frame number; and determine the to-be-consumed recycled frame number according to a difference between the total frame number of the frame memory pool and the queue frame number.

[0015] In an example embodiment of the present disclosure, the frame threshold acquisition module comprises a frame threshold acquisition unit configured to: acquire a device performance parameter of the media playback device, and a content feature parameter corresponding to the media content; and dynamically adjust the sent frame number threshold according to the device performance parameter and the content feature parameter.

[0016] In an example embodiment of the present disclosure, the frame output control module comprises a first frame output control unit configured to: when the to-be-consumed recycled frame number is greater than or equal to the sent frame number threshold, suspend a frame output operation of the codec to the rendering module.

[0017] In an example embodiment of the present disclosure, the first frame output control unit comprises a suspension output control subunit configured to: block a codec thread corresponding to the codec; reduce a thread priority corresponding to the codec thread; and reduce a codec rate corresponding to the codec.

[0018] In an example embodiment of the present disclosure, the frame output control module further comprises a second frame output control unit configured to: when the to-be-consumed recycled frame number is less than the sent frame number threshold, restore the frame output operation of the codec to the rendering module.

[0019] In an example embodiment of the present disclosure, the frame memory management apparatus further comprises a state identification module configured to: determine a frame life cycle state corresponding to each frame in each frame processing operation based on a frame state tracking mechanism, the frame processing operation comprising one or more of a frame decoding operation, a rendering consumption operation, and a frame recycling operation; and update a frame state identification according to the frame life cycle state, the frame life cycle state comprising a frame processing state, a frame consumption state, and a frame recycling state.

[0020] In an example embodiment of the present disclosure, the frame memory management apparatus further comprises a frame recycling module configured to: monitor a module running state of the rendering module; when the module running state is in an abnormal state, determine a to-be-recycled frame based on the frame life cycle state and a state duration; and perform a frame recycling operation on the to-be-recycled frame.

[0021] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the frame memory management method of any one of the above.

[0022] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the frame memory management method of any one of the above.

[0023] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the frame memory management method of any one of the above.

[0024] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0025] On the one hand, by controlling the number of frames to be consumed and recycled sent by the codec, the problem of memory accumulation and overflow caused by the fact that the subsequent frame processing module cannot process in time can be effectively avoided. On the other hand, in a resource-limited environment, the balance between the codec and the subsequent frame processing module can be automatically adjusted to ensure stable operation of the system and avoid system crash due to memory depletion. On the other hand, by dynamically controlling the frame output rate, the waste of resources caused by continuing to decode new frames when the subsequent frame processing module cannot process in time is avoided, and the utilization efficiency of CPU and memory resources is improved.

[0026] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0028] Figure 1 is a schematic diagram of frame memory management between a decoding module and a rendering module in a related technical solution.

[0029] Figure 2 is a flowchart of a frame memory management method according to an exemplary embodiment.

[0030] Figure 3 is a flowchart of a frame output operation of a decoding module based on a frame memory pool according to an exemplary embodiment.

[0031] Figure 4Fig. 1 is a schematic diagram illustrating a frame output operation control method according to a comparison result of a to-be-consumed frame number and a threshold of a sent frame number according to an example embodiment.

[0032] Figure 5 Fig. 2 is a block diagram of a frame memory management device according to an example embodiment.

[0033] Figure 6 Fig. 3 is a block diagram of an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0035] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following example embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0036] For frame memory management between a decoding module and a rendering module, the decoding module and the rendering module are usually two decoupled functional units, and the framework of frame output and frame consumption between the two is as shown in Fig. 1. Figure 1 Figure 1 Fig. 1 is a schematic diagram of frame memory management between a decoding module and a rendering module in a related technical solution. Based on this framework, there are the following three solutions:

[0037] Technical solution one: unlimited output solution. In technical solution one, the decoding module does not limit the number of output frames, and continuously delivers the decoded frames to the rendering module. However, technical solution one has the following defects: 1) risk of memory overflow. When the rendering module cannot process the decoded frames in time due to various reasons (such as excessive system load, rendering thread blocking, etc.), the frames that are not consumed will accumulate in the memory, eventually causing the memory usage to increase rapidly, causing the application program to crash due to Out Of Memory (OOM). 2) Low resource utilization efficiency. The decoding module may continue to decode new frames even if the rendering module cannot process existing frames in time, which will cause waste of computing resources.

[0038] ​Solution two: fixed buffer solution. A fixed size buffer is set between decoding and rendering. The decoder puts the decoded frame into the buffer, and the rendering module takes the frame from the buffer for display. Solution two has the defect of lacking dynamic adaptation capability. The fixed buffer solution is difficult to adapt to the dynamic demand changes of different devices and different media content.

[0039] Solution three: time-based throttling solution. According to the media playback rate, the decoding speed of the decoder is controlled to match the processing capability of the rendering module. Solution three is also prone to cause the problem of user experience degradation. When the system resources are tight, the cooperation relationship between decoding and rendering cannot be reasonably controlled, which may cause video playback to be stuck, delayed or crashed.

[0040] Based on this, according to the embodiments of the present disclosure, a frame memory management method, a frame memory management device, an electronic device, a computer readable storage medium and a computer program product are proposed.

[0041] Figure 2 is a flow chart of a frame memory management method according to an exemplary embodiment. As shown in Figure 2 The frame memory management method can be used in a computer device. The computer device described in the present disclosure can include mobile terminal devices such as mobile phones, tablet computers, notebook computers, palm computers, personal digital assistants (PDA), and fixed terminal devices such as desktop computers. The present exemplary embodiment is exemplified by the method applied to a computer device. It can be understood that the method can also be applied to a server, and can also be applied to a system including a computer device and a server, and is realized through the interaction of the computer device and the server. Specifically, the following steps are included.

[0042] In step S210, the number of frames to be consumed and recycled sent by the codec is determined, and the number of frames to be consumed and recycled is determined based on the total number of frames in the frame memory pool and the number of queue frames corresponding to the reference frame queue.

[0043] In step S220, a pre-configured number of sent frames threshold is obtained.

[0044] In step S230, the frame output operation of the codec is dynamically adjusted according to the comparison result of the number of frames to be consumed and recycled and the number of sent frames threshold.

[0045] According to the frame memory management method in the example embodiment, on one hand, by controlling the number of frames to be consumed and recycled sent by the codec, the problem of memory accumulation and overflow caused by the fact that the subsequent frame processing module cannot process in time can be effectively avoided. On the other hand, in the environment with limited resources, the balance between the codec and the subsequent frame processing module can be automatically adjusted to ensure the stable operation of the system and avoid the collapse caused by the exhaustion of memory. On the other hand, by dynamically controlling the frame output rate, the waste of resources caused by the fact that the subsequent frame processing module cannot process in time and the new frame continues to be decoded can be avoided, and the utilization efficiency of CPU and memory resources can be improved.

[0046] In the following, the frame memory management method in the example embodiment will be further described.

[0047] The frame memory management method in the disclosure is not only applicable to the decoder, but also applicable to the encoder. In order to facilitate understanding, the disclosure will take the application scene of the frame memory management method in the decoder as an example for specific explanation and description. The decoder in the disclosure can also be referred to as a decoding module.

[0048] In an example embodiment of the disclosure, for step S210, determining the number of frames to be consumed and recycled sent by the codec includes: creating a frame memory pool of the codec and configuring the total number of frames corresponding to the frame memory pool; taking the number of frames required by the reference frame queue in the codec as the number of queue frames; and determining the number of frames to be consumed and recycled according to the difference between the total number of frames of the frame memory pool and the number of queue frames.

[0049] The codec can be responsible for compressing (encoding) and decompressing (decoding) video data in video format conversion to realize file size reduction and compatible playback. The codec can be a technical module combined by an encoder and a decoder. The frame memory pool can be a memory pool created in the video processing architecture for temporarily storing video frames and frame-related data (such as data packets). The total number of frames corresponding to the frame memory pool can be an upper limit value of the total number of frames that can be accommodated in the frame memory pool, which can be represented by T. The reference frame queue can be a queue storing the reference frames currently required by the decoder, and the number of queue frames can be the number of frames required in the reference frame queue, which can be represented by N. The number of frames to be consumed and recycled can be the number of frames that have been sent to the subsequent frame processing module but have not been consumed and recycled by the decoder, which can be represented by S.

[0050] Reference Figure 3 , Figure 3 is a flowchart of controlling the frame output operation of the decoding module based on the frame memory pool according to an example embodiment. In order to solve the problem of the decoder outputting frames to the rendering module, the disclosure proposes to control the frame output operation of the decoding module based on the frame memory pool. Figure 1The architecture establishes a frame memory pool for storing frames flowing between different modules. For example, the upper limit value of the total number of frames corresponding to the frame memory pool is set as T.

[0051] In addition, in the decoder, the reference frame queue can be a decoded picture buffer (DPB), which is a very important component for storing reference frames in the decoding process. The number of frames required by the reference frame queue in the codec is set as N, which is used as the number of queue frames.

[0052] Then, according to the difference between the total number of frames T of the frame memory pool and the number of frames N in the queue, the number of frames that have been sent to the rendering module by the decoder but have not been consumed and recycled, i.e., the number of frames to be consumed and recycled S, is calculated, where S=T-N. By creating a frame memory pool and setting the upper limit of the total number of frames of the frame memory pool, the number of frames that have been sent to the rendering module by the decoder but have not been consumed and recycled can be determined according to the upper limit of the total number of frames and the number of frames in the reference frame queue, so as to be used as a judgment basis for whether the decoder continues to perform frame output operation. The present disclosure creates a frame memory pool, uses a new codec memory application and reuse scheme based on the technical scheme of the fast forward moving picture experts group (FFMPEG) integrated codec, so that the codec in FFMPEG avoids applying two frame memories, and saves the computational resource overhead of copying frame memory.

[0053] In an exemplary embodiment of the present disclosure, for step S220, the pre-configured sent frame number threshold is obtained, including: obtaining the device performance parameters of the media playback device and the content characteristic parameters corresponding to the media content; and dynamically adjusting the sent frame number threshold according to the device performance parameters and the content characteristic parameters.

[0054] In the present disclosure, the sent frame number threshold can also be set as one of the judgment factors for controlling whether the decoder continues to perform frame output operation; wherein the sent frame number threshold can be a pre-configured upper limit value for limiting the number of frames sent by the encoder, which can be represented by M.

[0055] For example, according to the device performance parameters of the media playback device and the content characteristic parameters corresponding to the media content, as the reference basis for setting the sent frame number threshold M. The media playback device can refer to a hardware device capable of reading, decoding and outputting various types of media content (such as audio, video, image, text, etc.), and its core function is to convert digital or analog media files into signals (such as video, sound) that can be perceived by humans. For example, the media playback device can include but is not limited to a streaming media player, a high-definition multimedia player, and a computer software player, etc.

[0056] The device performance parameter can be a parameter that determines the playing quality, compatibility and use experience of the media playing device. The device performance parameter can include, but is not limited to, general basic parameters, video playing core parameters, network and intelligent function parameters, and special device exclusive parameters, etc. The general basic parameters can include, but are not limited to, parameters of processors and chips, memory (RAM) and storage (ROM) parameters, interface types and quantities, etc. The video playing core parameters can include, but are not limited to, resolution, refresh rate, color gamut and color depth, video decoding format, etc. The network and intelligent function parameters can include, but are not limited to, network protocol support, intelligent system and application program (APP) compatibility, cache and loading speed, etc.

[0057] The media content can be the media content played by the media playing device, which can include, but is not limited to, video, image, etc. The content feature parameter can be a parameter that describes the inherent characteristics of the media content itself in space, time, texture, motion, etc. The content feature parameter can include, but is not limited to, spatial domain feature parameters (describing the static characteristics of a single frame picture), time domain feature parameters (describing the dynamic characteristics of multiple frame pictures), color and brightness feature parameters (describing the light color properties of the picture), content type feature parameters (describing the macroscopic scene properties of the video), and region characteristic parameters (describing the differentiated features of different regions in the frame), etc.

[0058] According to the device performance parameters of different media playing devices and the content feature parameters of the media content, the number of sent frames threshold M is dynamically adjusted, so that the frame output operation of the decoder can be adapted to various different application scenarios. For example, for some media playing devices with poor device performance, a smaller M value can be configured for them to reduce the possibility of playing lag as much as possible.

[0059] In an exemplary embodiment of the present disclosure, for step S230, the frame output operation of the codec is dynamically adjusted according to the comparison result of the number of frames to be consumed and the number of sent frames threshold, including: when the number of frames to be consumed is greater than or equal to the number of sent frames threshold, the frame output operation of the codec to the rendering module is paused.

[0060] The frame output operation can be an operation of the codec outputting frames to other frame processing modules. For example, at the decoder end, the frame output operation can be an operation of the decoder sending frames to the rendering module.

[0061] Reference Figure 4 , Figure 4is a schematic diagram of controlling frame output operation according to the comparison result of the number of frames to be consumed and the number of threshold frames sent according to an exemplary embodiment. In the frame processing process, the number of frames to be consumed S is monitored in real time, and the number of frames to be consumed S determined in real time is compared with the number of threshold frames sent M in step S410. When the number of frames to be consumed S is greater than or equal to the number of threshold frames sent M, it means that there are many frames sent out but not consumed or recycled in the true buffer queue between the decoder and the rendering module. At this time, step S420 needs to be executed to suspend the frame output operation of the codec to the rendering module. By suspending the frame output operation, the number of frames output by the decoder but not consumed can be limited, the system can be ensured to run stably, the risk of memory overflow can be avoided, and the resource utilization efficiency can be optimized.

[0062] In an exemplary embodiment of the present disclosure, suspending the frame output operation of the codec to the rendering module includes one or a combination of the following operations: blocking the codec thread corresponding to the codec; reducing the thread priority corresponding to the codec thread; and reducing the codec rate corresponding to the codec.

[0063] The codec thread can be an independent thread for coding and decoding video data. The codec thread can include an encoding thread and a decoding thread, such as creating an encoding thread in an encoder and creating a decoding thread in a decoder. The thread priority can be the priority corresponding to the codec thread. The codec rate can refer to the amount of data transmitted or processed per unit time.

[0064] At the decoder end, suspending the frame output operation of the decoder to the rendering module can be achieved by the following operations: blocking the decoding thread of the decoder; for example, by pausing a decoding thread during execution to wait for a condition to trigger its continued execution. Reducing the priority of the decoding thread; for example, different decoding threads can be configured with priorities when creating the decoding threads. In order to suspend the frame output operation of the decoder, the priority of the decoding thread can be reduced to delay its execution. Reducing the decoding rate of the decoder to control the decoder to suspend the frame output operation to the rendering module.

[0065] The dynamic control of the frame output operation of the codec to the rendering module can be achieved by one or a combination of the above operations to suspend the frame output operation of the decoder, so as to avoid the memory overflow problem caused by the rendering module due to the inability to perform frame processing operation in time.

[0066] In an exemplary embodiment of the present disclosure, for step S230, dynamically adjusting the frame output operation of the codec according to the comparison result of the number of frames to be consumed and the number of threshold frames sent further includes: when the number of frames to be consumed is less than the number of threshold frames sent, resuming the frame output operation of the codec to the rendering module.

[0067] Since the number of frames to be consumed and recycled S is monitored in real time during the frame processing process, the following is continued to be referred to Figure 4 When it is detected that the rendering module has completed consumption and recycled at least one frame, at this time the number of frames to be consumed and recycled S is less than the threshold number of frames M that have been sent, it is considered that the frame buffer queue can store the frames output by the decoder, at this time step S430 can be executed to restore the normal decoding and output operation of the decoding module. By dynamically monitoring and limiting the number of frames that have been output but not consumed, it is possible to dynamically control whether to perform frame output operation, so as to ensure that the system can stably run under various load conditions, avoid the risk of memory overflow, and at the same time optimize the resource utilization efficiency.

[0068] In an exemplary embodiment of the present disclosure, based on a frame state tracking mechanism, the frame lifecycle state corresponding to each frame in each frame processing operation is determined, the frame processing operation including one or more of a frame decoding operation, a rendering consumption operation, and a frame recycling operation; and according to the frame lifecycle state, the frame state identifier is updated, the frame lifecycle state including a frame processing state, a frame consumption state, and a frame recycling state.

[0069] The frame decoding operation can be an operation of decoding the frame by the decoder. The rendering consumption operation can be an operation of rendering the frame output by the decoding module by the rendering module. The frame recycling operation can be an operation of recycling the memory of the frame output by the decoding module but not rendered for a long time by the rendering module. The frame lifecycle state can be a lifecycle state that can be corresponded to by a certain video frame. The frame state identifier can be an identifier for indicating the lifecycle state currently corresponded to by a frame. The frame processing state can be a state identifier for indicating whether a frame has been processed by a codec. The frame consumption state can be a state identifier for indicating whether a frame has been rendered and consumed by the rendering module. The frame recycling state can be a state identifier for indicating whether a frame has been recycled.

[0070] The present disclosure adopts a frame state tracking mechanism to record the full lifecycle state of each frame from decoding completion to rendering consumption and recycling. Through the frame state tracking mechanism, the frame lifecycle state corresponding to each frame in each different frame processing operation can be determined, for example, in the process of outputting the frame by the decoder to the rendering module, the frame processing operation can include but is not limited to the frame decoding operation, the rendering consumption operation, and the frame recycling operation.

[0071] The frame state tracking mechanism can determine the corresponding frame life cycle state according to different frame processing operations undergone by each frame. For example, the frame life cycle state of a frame can include, but is not limited to, a frame processing state, a frame consumption state, and a frame recycling state, which are used to indicate whether the frame has been subjected to a decoding operation, whether the frame has been subjected to a rendering operation by the rendering module, or whether the frame has been recycled after being not consumed by the rendering module for a long time. By updating the frame state identifier according to the currently identified frame life cycle state, it can be determined that the frame is currently in which state. The state identifier can be used as a basis for determining whether to recycle a frame.

[0072] In an exemplary embodiment of the present disclosure, a module running state of the rendering module is monitored. When the module running state is in an abnormal state, a frame to be recycled is determined based on the frame life cycle state and the state duration. A frame recycling operation is performed on the frame to be recycled.

[0073] The module running state can be a state of whether the rendering module is normally running. The state duration can be a duration of a frame being in a certain life cycle state. The frame to be recycled can be a frame to be subjected to a memory recycling process. The frame recycling operation can be an operation of recycling a frame and memory occupied by the frame.

[0074] In order to recycle a frame that has been sent out by the decoder but has not been consumed by the rendering module in time, the present disclosure also supports an exception handling mechanism. The exception handling mechanism detects a module running state of the rendering module. When it is detected that the module running state of the rendering module is in an abnormal state, the frame life cycle state of a frame in the frame buffer queue is analyzed. If a frame life cycle state is in a to-be-consumed state, i.e., a state of waiting for the rendering module to perform rendering consumption, and the state duration of being in the to-be-consumed state has exceeded a pre-configured state duration threshold, the frame is determined to be a frame to be recycled. A frame recycling operation is performed on the frame to be recycled to prevent memory resources from being occupied for a long time.

[0075] To sum up, the frame memory management method of the present disclosure determines the number of frames to be consumed and recycled sent by the codec, the number of frames to be consumed and recycled is determined based on the total number of frames of the frame memory pool and the number of queue frames corresponding to the reference frame queue; a pre-configured threshold value of the number of sent frames is obtained; and the frame output operation of the codec is dynamically adjusted according to the comparison result of the number of frames to be consumed and recycled and the threshold value of the number of sent frames. On the one hand, by controlling the number of frames to be consumed and recycled sent by the codec, the problem of memory accumulation and overflow caused by the fact that the subsequent frame processing module cannot process in time can be effectively avoided. On the other hand, in a resource-limited environment, the balance between the codec and the subsequent frame processing module can be automatically adjusted to ensure stable operation of the system and avoid collapse due to memory depletion. On the other hand, by dynamically controlling the frame output rate, the waste of resources caused by continuing to decode new frames when the subsequent frame processing module cannot process in time is avoided, and the utilization efficiency of CPU and memory resources is improved. On the other hand, the application is prevented from freezing or collapsing due to memory problems, the smoothness and stability of media playback are ensured, and the user experience can be effectively improved.

[0076] Figure 5 is a frame memory management device block diagram according to an exemplary embodiment. Referring to Figure 5 The frame memory management device 500 comprises a frame number determination module 510, a frame threshold value acquisition module 520, and a frame output control module 530.

[0077] Specifically, the frame number determination module 510 is configured to determine the number of frames to be consumed and recycled sent by the codec, and the number of frames to be consumed and recycled is determined based on the total number of frames of the frame memory pool and the number of queue frames corresponding to the reference frame queue; the frame threshold value acquisition module 520 is configured to obtain a pre-configured threshold value of the number of sent frames; and the frame output control module 530 is configured to dynamically adjust the frame output operation of the codec according to the comparison result of the number of frames to be consumed and recycled and the threshold value of the number of sent frames.

[0078] In an exemplary embodiment of the present disclosure, the frame number determination module 510 comprises a frame number determination unit configured to: create a frame memory pool of the codec, and configure the total number of frames corresponding to the frame memory pool; take the number of frames required by the reference frame queue in the codec as the number of queue frames; and determine the number of frames to be consumed and recycled according to the difference between the total number of frames of the frame memory pool and the number of queue frames.

[0079] In an exemplary embodiment of the present disclosure, the frame threshold value acquisition module 520 comprises a frame threshold value acquisition unit configured to: obtain the device performance parameters of the media playback device and the content feature parameters corresponding to the media content; and dynamically adjust the threshold value of the number of sent frames according to the device performance parameters and the content feature parameters.

[0080] In an example implementation of the present disclosure, the frame output control module 530 comprises a first frame output control unit configured to: suspend the frame output operation of the codec to the rendering module when the number of frames to be consumed and recycled is greater than or equal to the threshold of the number of frames sent.

[0081] In an example implementation of the present disclosure, the first frame output control unit comprises a suspension output control subunit configured to: block the codec thread corresponding to the codec; reduce the thread priority corresponding to the codec thread; and reduce the codec rate corresponding to the codec.

[0082] In an example implementation of the present disclosure, the frame output control module 530 further comprises a second frame output control unit configured to: resume the frame output operation of the codec to the rendering module when the number of frames to be consumed and recycled is less than the threshold of the number of frames sent.

[0083] In an example implementation of the present disclosure, the frame memory management apparatus 500 further comprises a state identification module configured to: determine the frame lifecycle state corresponding to each frame in each frame processing operation based on the frame state tracking mechanism, the frame processing operation comprising one or more of the frame decoding operation, the rendering consumption operation, and the frame recycling operation; and update the frame state identification according to the frame lifecycle state, the frame lifecycle state comprising the frame processing state, the frame consumption state, and the frame recycling state.

[0084] In an example implementation of the present disclosure, the frame memory management apparatus 500 further comprises a frame recycling module configured to: monitor the module running state of the rendering module; determine the frame to be recycled based on the frame lifecycle state and the state duration when the module running state is in an abnormal state; and perform the frame recycling operation on the frame to be recycled.

[0085] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0086] The electronic device 600 according to this embodiment of the present disclosure will be described below with reference to Figure 6 Figure 6 The display electronic device 600 is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0087] As Figure 6 shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include, but are not limited to, the above-mentioned at least one processing unit 610, the above-mentioned at least one storage unit 620, a bus 630 connecting different system components including the storage unit 620 and the processing unit 610, and a display unit 640.

[0088] ​The storage unit stores program codes which can be executed by the processing unit 610, so that the processing unit 610 performs the steps described in the above "Exemplary Method" section according to various exemplary embodiments of the present disclosure.

[0089] The storage unit 620 can include a readable medium in the form of volatile storage such as random access memory (RAM) 621 and / or cache memory 622, and further can include a non-volatile storage such as read-only memory (ROM) 623.

[0090] The storage unit 620 can include program / utility 624 having a set of at least one program modules 625, including but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, which can include implementation of a network environment.

[0091] The bus 630 can represent one or more of several types of bus structures, including a storage bus or bus controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures.

[0092] The electronic device 600 can also communicate with one or more external devices 670 such as a keyboard or pointing device, a Bluetooth device, etc.; other devices such as devices enabling a user to interact with the electronic device 600; and / or one or more devices enabling the electronic device 600 to communicate with one or more other computing devices. Such communication can be facilitated by an Input / Output (I / O) interface 650. Still yet, the electronic device 600 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the public network, such as the Internet, via a network adapter 660. As depicted, the network adapter 660 is in communication with the other components of the electronic device 600 through the bus 630. It should be appreciated that although not shown, other hardware and / or software components could be used in conjunction with the electronic device 600. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0093] In exemplary embodiments, a computer readable storage medium including instructions, such as a memory including instructions, is also provided, which can be executed by a processor of a device to complete the above-mentioned frame memory management method. Alternatively, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0094] In an exemplary embodiment, a computer program product is also provided, comprising a computer program which, when executed by a processor, implements the frame memory management method of any of the above.

[0095] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the concepts disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0096] It is to be understood that the disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the disclosure should only be limited by the appended claims.

Claims

1. An in-frame memory management method, characterized by, The method comprises the following steps: determining the number of frames to be consumed and recycled sent by the codec, which is determined based on the total number of frames in the frame memory pool and the number of frames corresponding to the reference frame queue; obtaining a pre-configured threshold value of the number of sent frames; dynamically adjusting the frame output operation of the codec according to the comparison result of the number of frames to be consumed and recycled and the threshold value of the number of sent frames.

2. The method of claim 1, wherein, The method comprises the following steps: creating a frame memory pool of the codec and configuring the total number of frames corresponding to the frame memory pool; taking the number of frames required by the reference frame queue in the codec as the number of queue frames; determining the number of frames to be consumed and recycled according to the difference between the total number of frames in the frame memory pool and the number of queue frames.

3. The method of claim 1, wherein, The method comprises the following steps: obtaining the device performance parameters of the media playback device and the content feature parameters corresponding to the media content; dynamically adjusting the threshold value of the number of sent frames according to the device performance parameters and the content feature parameters.

4. The method of claim 1, wherein, The method comprises the following steps: when the number of frames to be consumed and recycled is greater than or equal to the threshold value of the number of sent frames, pausing the frame output operation of the codec to the rendering module.

5. The method of claim 4, wherein, The method comprises the following steps: one or more combinations of the following operations: blocking the codec thread corresponding to the codec; reducing the thread priority corresponding to the codec thread; 6. The method of claim 4, wherein, reducing the codec rate corresponding to the codec. The method further comprises the following steps:

7. The method of claim 1, wherein, when the number of frames to be consumed and recycled is less than the threshold value of the number of sent frames, resuming the frame output operation of the codec to the rendering module. The method further comprises the following steps: based on the frame state tracking mechanism, determining the frame life cycle state of each frame in each frame processing operation, the frame processing operation comprising one or more of frame decoding operation, rendering consumption operation and frame recycling operation; 8. The method of claim 7, wherein, updating the frame state identifier according to the frame life cycle state, the frame life cycle state comprising frame processing state, frame consumption state and frame recycling state. The method further comprises the following steps: monitoring the module running state of the rendering module; when the module running state is in an abnormal state, determining the frame to be recycled based on the frame life cycle state and the state duration; 9. An in-frame memory management device, characterized by, performing frame recycling operation on the frame to be recycled. The method comprises the following steps: a frame number determination module for determining the number of frames to be consumed and recycled sent by the codec, which is determined based on the total number of frames in the frame memory pool and the number of frames corresponding to the reference frame queue; a frame threshold value acquisition module for obtaining a pre-configured threshold value of the number of sent frames; 10. An electronic device, comprising: a frame output control module for dynamically adjusting the frame output operation of the codec according to the comparison result of the number of frames to be consumed and recycled and the threshold value of the number of sent frames. The method comprises the following steps: a processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the frame memory management method according to any one of claims 1-8.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the frame memory management method according to any one of claims 1-8.

Images