Video decoding method suitable for embedded system

Abstract

The invention relates to a video decoding method suitable for an embedded system. The method comprises the following steps: for each acquired to-be-processed video decoding task, analyzing the video decoding task to obtain a coding parameter of a video stream; generating a task feature tag corresponding to the video decoding task according to the coding parameter and task source information, task type and task demand information carried in the video decoding task; according to the task feature tag corresponding to each video decoding task, performing priority classification on each video decoding task to be processed; detecting a decoding resource, a data transmission resource and load information of an image output module in the current system to obtain a current system resource state; according to a current system resource state, a hardware decoding channel is dynamically and preferentially allocated to a high-priority real-time task, and a software decoding channel is dynamically and preferentially allocated to a low-priority background task. According to the method, the video decoding performance can be improved.

Description

Technical Field

[0001] The present application relates to the field of embedded technology, and in particular to a video decoding method applicable to embedded systems. Background Art

[0002] With the rapid development of embedded devices such as smart terminals, IoT devices, in-vehicle systems, and intelligent surveillance, multimedia video decoding has become a key performance bottleneck in embedded systems. Embedded devices typically have limited resources and must meet the decoding requirements of high-definition video, multiple video streams, and heterogeneous multi-tasking within limited computing power and energy constraints.

[0003] Traditional solutions in embedded devices generally use fixed decoding resource allocation strategies. For example, decoding resources are typically allocated to video decoding tasks based on the order in which they are submitted. This approach is overly restrictive and cannot handle complex multi-task video scenarios. Consequently, decoding performance for critical tasks (such as real-time video conferencing) in concurrent multi-task scenarios is poor, impacting the user experience. Summary of the Invention

[0004] Based on this, it is necessary to provide a video decoding method, apparatus, embedded device, computer-readable storage medium and computer program product suitable for embedded systems that can improve decoding performance in response to the above technical problems.

[0005] In a first aspect, the present application provides a video decoding method applicable to an embedded system, the method comprising:

[0006] For each acquired video decoding task to be processed, the video decoding task is parsed to obtain encoding parameters of the video stream;

[0007] Generate a task feature label corresponding to the video decoding task according to the encoding parameters, the task source information, the task type, and the task requirement information carried in the video decoding task;

[0008] Prioritizing each of the video decoding tasks to be processed according to the task feature label corresponding to each of the video decoding tasks;

[0009] Detecting the decoding resources, data transmission resources, and load information of the image output module in the current system to obtain the current system resource status; the image output module is the module to which the decoded frame is output after decoding;

[0010] According to the current system resource status, hardware decoding channels are preferentially allocated to the first type of video decoding tasks, and software decoding channels are preferentially allocated to the second type of video decoding tasks; wherein, the first type of video decoding tasks are video decoding tasks classified as high-priority real-time tasks, and the second type of video decoding tasks are video decoding tasks classified as low-priority background tasks.

[0011] In one embodiment, the decoding resources include the number of idle channels of the hardware decoding unit, the encoding formats supported by the hardware decoding unit, and the schedulable computing resources of the software decoding module.

[0012] In one embodiment, the data transmission resources include an idle direct memory access channel, a data sharing mechanism with the image output module, and a current transmission bandwidth.

[0013] In one embodiment, the load information of the image output module includes information representing the current processing capability of the image output module.

[0014] In one embodiment, the task feature tag includes at least one of a real-time level tag, a fault tolerance level, degradability, a time priority level, a resource dependency strength, a flow feature tag, or a processing chain dependency.

[0015] In one embodiment, the real-time level label includes high real-time, medium real-time or low real-time; the fault tolerance level includes strict, medium or loose; the degradability is used to characterize whether it supports reduced resolution and / or frame rate processing; the level of the time priority is positively correlated with the strength of time sensitivity; the resource dependency strength is used to characterize whether it is strongly dependent on hardware decoding resources, and / or whether shared buffer resources are required; the stream feature label includes long continuous, short burst, high bandwidth or low bit rate; the processing chain dependency is used to characterize whether it needs to flow into the image output module immediately after decoding.

[0016] In one embodiment, the method further comprises:

[0017] Binding a corresponding shared buffer to the hardware decoding channel and / or the software decoding channel;

[0018] After the video is decoded, the decoded frame is stored in the bound shared buffer; the decoded frame refers to the image frame after the video stream is decoded;

[0019] The decoded frame in the shared buffer is directly transmitted to the image output module through physical address mapping or buffer reference.

[0020] In one embodiment, the obtained classification category further includes a medium-priority quasi-real-time task; the real-time requirement of the medium-priority quasi-real-time task is lower than that of the high-priority real-time task, but higher than that of the low-priority background task; the video decoding task classified as the medium-priority quasi-real-time task is a third-category video decoding task;

[0021] The method further comprises:

[0022] After hardware decoding channels are allocated to the first type of video decoding tasks, if idle hardware decoding channels still exist, the hardware decoding channels are preferentially allocated to the third type of video decoding tasks.

[0023] In one embodiment, the method further comprises:

[0024] For a first target task, if resources of the hardware decoding channel are insufficient or decoding using the hardware decoding channel fails, switching allocation of the software decoding channel to the first target task; the first target task includes the first type of video decoding task and / or the third type of video decoding task;

[0025] In one embodiment, the method further comprises:

[0026] For the second target task, when the data transmission resources are insufficient, reducing the image resolution or frame rate required by the second target task, the second target task including the second type of video decoding task and / or the third type of video decoding task;

[0027] In one embodiment, the method further comprises:

[0028] When the load of the image output module is saturated, the video decoding task with a lower priority is cached and waits.

[0029] In one embodiment, detecting the load information of decoding resources, data transmission resources, and image output modules in the current system to obtain the current system resource status includes:

[0030] Call the hardware acceleration interface to detect the decoding resources, data transmission resources and image output module load information in the current system to obtain the current system resource status.

[0031] In a second aspect, the present application further provides a video decoding device suitable for an embedded system, the device comprising:

[0032] A task parsing module is used to parse each acquired video decoding task to be processed and obtain encoding parameters of the video stream;

[0033] A task feature generation module is used to generate a task feature label corresponding to the video decoding task based on the encoding parameters, the task source information, the task type and the task requirement information carried in the video decoding task;

[0034] A priority classification module, configured to classify the priorities of the video decoding tasks to be processed according to the task feature labels corresponding to the video decoding tasks;

[0035] A resource status detection module is used to detect the load information of the decoding resources, data transmission resources and image output module in the current system to obtain the current system resource status; the image output module is the module to which the decoded frame is output after decoding;

[0036] A decoding channel allocation module is used to prioritize the allocation of hardware decoding channels for the first category of video decoding tasks and prioritize the allocation of software decoding channels for the second category of video decoding tasks based on the current system resource status; wherein the first category of video decoding tasks is a video decoding task classified as a high-priority real-time task, and the second category of video decoding tasks is a video decoding task classified as a low-priority background task.

[0037] In a third aspect, the present application further provides an embedded device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method mentioned in the first aspect.

[0039] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method mentioned in the first aspect.

[0040] The above-mentioned video decoding method, apparatus, embedded device, computer-readable storage medium, and computer program product for embedded systems analyzes the task feature label for each video decoding task to be processed, combining the encoding parameters of the video stream indicated by the task, the task source information carried in the video decoding task, the task type, and the task requirement information, and other multi-dimensional information. Based on the task feature label, the video decoding task is dynamically and more accurately prioritized. Compared with allocating decoding resources according to the order in which tasks are submitted, the subsequent priority based on dynamic analysis can more accurately allocate the corresponding decoding resources. In addition, in addition to combining the priority of dynamic analysis, it also combines the system resource status detection. Based on the priority of dynamic analysis and the current system resource status, the accuracy of decoding resource allocation can be further improved. Moreover, when allocating decoding resources, hardware decoding resources and software decoding resources can be dynamically and reasonably selected based on the analysis results. In this way, video decoding tasks with real-time requirements and higher priority can be preferentially allocated with hardware decoding resources to meet their requirements for decoding performance. Low-priority tasks that don't require high hardware decoding resources can be prioritized for software decoding resources, reducing the use of hardware decoding resources. This allows for more efficient allocation of decoding resources and improves resource utilization. This dynamic scheduling and allocation of decoding resources is particularly suitable for complex, multi-tasking video scenarios, improving overall decoding performance and accelerating decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 1 is a flow chart of a video decoding method applicable to an embedded system in one embodiment;

[0042] Figure 2 is a flow chart of a video decoding method applicable to an embedded system in another embodiment;

[0043] Figure 3 is a structural block diagram of a video decoding device suitable for an embedded system in one embodiment;

[0044] Figure 4 is a structural block diagram of a video decoding device suitable for an embedded system in another embodiment;

[0045] Figure 5 FIG. 4 is a diagram showing the internal structure of an embedded device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] In one embodiment, a video decoding method applicable to an embedded system is provided. The embedded system is provided with hardware support by an embedded device, and the method is executed by the embedded device. Figure 1 As shown, the method specifically includes the following steps:

[0048] S11 , for each acquired video decoding task to be processed, parsing the video decoding task to obtain encoding parameters of the video stream.

[0049] It should be understood that in a multi-tasking concurrent scenario, during the system operation of an embedded device, multiple video decoding tasks will be received from upper-layer applications. Among them, a video decoding task refers to a task involving video decoding.

[0050] Specifically, an upper-layer application in an embedded system (such as a local player or a streaming media receiver) issues a video decoding task through a system interface.

[0051] Exemplarily, a video decoding task includes at least one of: task source information, task type, or task requirement information. Task source information refers to the location of the video source data, such as a local media file path or a network stream address. Task types include at least one of preview, storage, and recognition. Task requirement information includes at least one of whether the task is used in a real-time scenario (i.e., real-time requirements) or whether degradation processing is allowed.

[0052] For each video decoding task to be processed, steps S11 to S14 can be executed, and then corresponding decoding resources can be adaptively and dynamically allocated to each video decoding task through step S15 to improve resource utilization, thereby improving the overall decoding performance of the system.

[0053] Specifically, for each video decoding task, the embedded device can decapsulate the video stream and perform parameter preprocessing to extract the encoding parameters of the video stream. For example, the encoding parameters include key technical indicators such as encoding format (such as H.264, H.265), image resolution, frame rate, and bit rate.

[0054] In some examples, embedded devices can demultiplex video sources and extract parameters through the FFmpeg interface. Specifically, the media input can be opened through the avformat_open_input() function, and the video stream information can be analyzed through the avformat_find_stream_info() function. Based on the analysis results of the video stream, the encoding parameters can be extracted through the avcodec_parameters_to_context() function and the decoding context can be configured.

[0055] S12: Generate a task feature label corresponding to the video decoding task according to the encoding parameters, the task source information, the task type, and the task requirement information carried in the video decoding task.

[0056] Specifically, the embedded device may analyze and evaluate the processing characteristics of the video decoding task, such as at least one of real-time requirements, interaction sensitivity, or fault tolerance, based on the encoding parameters, task source information, task type, and task requirement information carried in the video decoding task. Based on the analyzed processing characteristics, the embedded device may generate a task feature label corresponding to the video decoding task.

[0057] Exemplarily, the encoding parameters, task source information, task type and task requirement information can be used to generate corresponding vector representations and input into a pre-trained task feature generation model. According to the task feature generation model, the vector representations of the above multiple dimensions are subjected to feature convolution and extraction under preset feature dimensions to generate corresponding task feature labels under each preset feature dimension.

[0058] In some embodiments, the task feature tag includes at least one of a real-time level tag, a fault tolerance level, degradability, a time priority level, a resource dependency strength, a flow feature tag, or a processing chain dependency.

[0059] In some examples, the real-time level label includes high real-time (such as a camera preview task), medium real-time (such as a recording and playback task), or low real-time (such as an offline transcoding task).

[0060] In some examples, the fault tolerance level includes strict (such as video decoding tasks in industrial visual recognition scenarios), medium (such as video decoding tasks in security monitoring scenarios), or loose (such as background storage).

[0061] In some examples, the degradability is used to indicate whether processing with reduced resolution and / or frame rate is supported.

[0062] In some examples, the time priority level is positively correlated with the time sensitivity. For example, the time priority level can be divided into high, medium, and low priority levels according to time sensitivity and / or service type.

[0063] In some examples, the resource dependency strength is used to characterize whether there is a strong dependence on hardware decoding resources and / or whether shared buffer resources are required, for example, whether DMA (Direct Memory Access) is required, or whether a zero-copy mechanism is required.

[0064] In some examples, the flow feature tags include long-term continuous, short-term burst, high bandwidth or low bit rate, etc.

[0065] In some examples, the processing chain dependency is used to indicate whether the decoded frame needs to be immediately fed into the image output module. The image output module is the module to which the decoded frame is output after decoding. For example, the image output module includes at least one of an image display interface, a video re-encoding module, or an image analysis engine (i.e., a post-processing engine).

[0066] S13, classifying the priority of each of the video decoding tasks to be processed according to the task feature label corresponding to each of the video decoding tasks.

[0067] For the real-time level label in the task feature label, the priority is positively correlated with the real-time level, that is, the higher the real-time requirement, the higher the priority, and vice versa, the lower the real-time requirement, the lower the priority.

[0068] For the fault tolerance level in the task feature label, the stricter the fault tolerance level, the higher the priority, and the looser the fault tolerance level, the lower the priority.

[0069] For the degradability in the task feature tag, if degradation processing is supported, the priority is lower; if degradation processing is not supported, the priority is higher.

[0070] For the time priority level in the task feature label, there is a positive correlation between the priority level and the time priority level.

[0071] For the resource dependency strength in the task feature label, the priority is positively correlated with the resource dependency strength. For example, the more dependent on hardware decoding resources, the higher the priority.

[0072] For the stream feature tags in the task feature tags, the priority is determined by the specific resource requirements. For example, if a low bitrate is required, the priority is low.

[0073] For the processing chain dependency in the task feature tag, the priority is positively correlated with the processing chain dependency. For example, if the image output module needs to be immediately fed into the decoded image, the stronger the processing chain dependency, the higher the priority.

[0074] The above describes the relationship between the priority and the specific features under a single feature dimension. However, the task feature label has multiple dimensions (i.e., multiple feature dimensions). For example, the real-time level label and the stream feature label belong to different feature dimensions. When determining the priority of a certain video decoding task, the priority or importance of different feature dimensions are different. For example, real-time requirements > resolution requirements > frame rate requirements > bit rate requirements. It should be understood that the real-time requirement is determined by the real-time level label in the task feature label, and the requirements such as resolution, frame rate, and bit rate can be determined by the stream feature label, which is equivalent to the importance of the real-time level label being greater than the importance of the stream feature label.

[0075] In some examples, corresponding weights can be assigned to the task feature labels of each dimension. It should be understood that for each feature dimension, a corresponding mapping relationship can be preset to map the specific task feature labels under the feature dimension to the corresponding priority scores. Then, according to the preset weights, the priority scores under each feature dimension (i.e., the task feature labels of each dimension) are weighted and summed to obtain the total score corresponding to each video decoding task. Furthermore, the total scores corresponding to each video decoding task are used to determine the classification category of the priority corresponding to each video decoding task.

[0076] In some embodiments, the classification category includes at least one of a high priority real-time task, a medium priority near real-time task, or a low priority background task.

[0077] High-priority real-time tasks are high-priority video decoding tasks with strict real-time requirements. These tasks are classified as first-class video decoding tasks. Examples include video preview, real-time monitoring, and low-latency analysis.

[0078] Medium-priority quasi-real-time tasks are video decoding tasks that have a certain degree of real-time performance but can tolerate slight delays. These tasks are classified as the third category. Examples include simultaneous recording and playback and delayed streaming.

[0079] Low-priority background tasks are video decoding tasks with low real-time requirements. These tasks are classified as Category 2 video decoding tasks. Examples include offline transcoding, storage, and non-interactive analysis.

[0080] It should be understood that "first", "second", "third", etc. in the embodiments of the present application are only used for distinction and are not used to limit relationships such as size, sequence, subordination, and level.

[0081] S14, detecting the load information of the decoding resources, data transmission resources and image output module in the current system to obtain the current system resource status.

[0082] In some examples, the decoding resources include hardware decoding resources and software decoding resources. Hardware decoding resources may include the number of idle channels of the hardware decoding unit and the encoding formats supported by the hardware decoding unit. Software decoding resources may include the schedulable computing resources of the software decoding module (e.g., CPU core occupancy, temperature constraints, etc.).

[0083] In some embodiments, the embedded device can call a hardware acceleration interface (such as VAAPI, Video Acceleration API, a set of interface standards for video processing hardware acceleration) to detect the hardware decoding resources in the current system. For example, the vaQueryConfigProfiles() and vaQueryConfigEntrypoints() functions can be used to analyze whether the hardware decoding unit supports the encoding format, and the vaCreateConfig() and vaCreateContext() functions can be used to verify whether the hardware decoding channel is available or idle, thereby obtaining the hardware decoding resources.

[0084] In some examples, the data transmission resource includes at least one of an idle direct memory access channel (i.e., a DMA channel), a data sharing mechanism with the image output module, or a current transmission bandwidth. Specifically, it can be detected whether there is an idle or allocable DMA channel, and it can be determined whether there is a direct data sharing mechanism (e.g., a shared buffer, a physically continuous memory mapping area, etc.) between the decoding unit and the image output module.

[0085] In some examples, the image output module load information includes information representing the current processing capability of the image output module (i.e., information representing the current processing capability of the image output module). The current processing capability of the image output module refers to the ability to accept and process new frames. It should be understood that evaluating the image output module load based on the current processing capability of the image output module can avoid subsequent data congestion or frame loss, thereby improving task execution.

[0086] It should be understood that the detected decoding resources, data transmission resources and image output module load information in the current system constitute the current system resource status, which can be used for subsequent task scheduling and decoding channel (or decoding path) selection stages.

[0087] S15. According to the current system resource status, hardware decoding channels are preferentially allocated to the first type of video decoding tasks, and software decoding channels are preferentially allocated to the second type of video decoding tasks; wherein, the first type of video decoding tasks are video decoding tasks classified as high-priority real-time tasks, and the second type of video decoding tasks are video decoding tasks classified as low-priority background tasks.

[0088] After priority classification, tasks can be scheduled based on the classification categories in combination with the current system resource status. Specifically, the embedded device selects the appropriate decoding channel (or decoding path) for each video decoding task based on the task scheduling rules.

[0089] In some examples, the task scheduling rules include at least one of the following: giving priority to tasks with high real-time requirements (such as preview tasks, low-latency analysis tasks, etc.); reserving available hardware decoding resources for high-priority tasks; using software decoding or low frame rate processing for delayable tasks (such as recording, non-interactive analysis); if resources are tight, setting task queuing, merging, or degradation strategies.

[0090] In some embodiments, since the first category of video decoding tasks is a high-priority real-time task with the highest real-time requirement and the highest priority, the embedded device can prioritize the allocation of hardware decoding channels for the first category of video decoding tasks and prioritize the first category of video decoding tasks. Specifically, if there are idle hardware decoding resources in the system and the task requirements are adapted to the codec format and resolution supported by the hardware, the first category of video decoding tasks will be allocated to the hardware decoding channel. For example, if the VAAPI decoder is available (i.e., there are available hardware decoding resources), a hardware decoding channel will be allocated to the video decoding task classified as a high-priority real-time task.

[0091] In some embodiments, as described above, the third type of video decoding task is a medium-priority quasi-real-time task. It should be understood that the real-time requirements of a medium-priority quasi-real-time task are lower than those of the high-priority real-time task, but higher than those of the low-priority background task. Therefore, if a hardware decoding channel is still available after allocating it to the first type of video decoding task, the hardware decoding channel will be preferentially allocated to the third type of video decoding task.

[0092] In some embodiments, the second type of video decoding task is a low-priority background task. Since its priority is relatively low and the real-time requirements are relatively low, the software decoding channel is preferentially allocated to the second type of video decoding task, thereby avoiding tasks with lower real-time requirements occupying hardware decoding resources, making resource allocation more reasonable and improving resource utilization.

[0093] In the above method, for each pending video decoding task, task feature tags are analyzed based on multiple dimensions of information, including the encoding parameters of the video stream indicated by the task, the task source information carried by the task, the task type, and task requirement information. Based on the task feature tags, video decoding tasks are dynamically and more accurately prioritized. Compared to allocating decoding resources based on the order in which tasks are submitted, subsequent allocation of corresponding decoding resources based on dynamically analyzed priorities is more accurate. Furthermore, in addition to incorporating dynamic analysis priorities, system resource status detection is also incorporated. Based on the dynamic analysis priorities and the current system resource status, the accuracy of decoding resource allocation can be further improved. Furthermore, when allocating decoding resources, hardware and software decoding resources can be dynamically and appropriately selected based on the analysis results. This allows video decoding tasks with high real-time requirements and high priority to be prioritized with hardware decoding resources, meeting their decoding performance requirements. Low-priority tasks with low hardware decoding resource requirements can be prioritized with software decoding resources, thereby reducing hardware decoding resource usage, achieving more reasonable decoding resource allocation, and improving resource utilization. This improves overall decoding performance and achieves decoding acceleration. It should be understood that the above-mentioned solution of dynamically scheduling and allocating decoding resources is particularly suitable for resource-constrained embedded multimedia platforms and meets the real-time and stability requirements in complex multi-channel video processing scenarios.

[0094] In addition, this application is a brand-new solution specifically proposed for video decoding scenarios in embedded devices. It fully analyzes and studies the unique task characteristics of video decoding scenarios in embedded systems, deeply perceives the underlying resource status of various video decoding related to embedded systems, and dynamically binds decoding channels based on the analyzed features and the deeply perceived underlying resource status. The decoding channels are not fixed but can be dynamically adjusted in real time. This can more reasonably achieve dynamic allocation of decoding resources and improve decoding performance. Moreover, when detecting resource status, the timeliness of consumption of decoded frames in subsequent display, analysis and other image output modules is also taken into account, further improving the overall performance of the decoding process.

[0095] It should be understood that video decoding involves reading and writing large amounts of data in memory and transferring data between the processor and the hardware decoding module. Traditional data transmission methods rely heavily on the CPU for data copying and management, resulting in excessive CPU load and limiting the room for improving decoding efficiency. In some embodiments of the present application, direct memory access (DMA) technology is combined to enable the image output module to directly access system memory, bypassing the CPU for data transfer, thereby reducing CPU intervention and improving data transmission speed and efficiency. Using DMA technology to achieve data sharing can effectively reduce system latency and energy consumption.

[0096] Specifically, in some embodiments, after allocating a corresponding decoding channel (or decoding path) to each video decoding task, the embedded device binds a corresponding data buffer to each video decoding task, and selects a shared memory area that supports DMA or zero-copy mechanism as much as possible to reduce system bus transmission and CPU intervention, thereby significantly reducing the CPU participation and system load.

[0097] Exemplarily, for the hardware decoding channel and / or the software decoding channel, corresponding shared buffers are bound respectively. The shared buffers can be created through a DMA interface.

[0098] It should be understood that the hardware decoding channel is bound to a hardware shared buffer, and the software decoding channel is bound to a software shared buffer. For example, when the hardware decoding channel is bound, the hardware shared buffer for output can be allocated for the decoded frame through the vaCreateSurfaces() function in the hardware acceleration interface (VAAPI).

[0099] In some embodiments, after the corresponding shared buffer is bound to the decoding channel, the video decoding task enters the actual decoding phase. In the decoding phase, the decoding module decodes the encoded video stream into image frames, and the decoded image frames are decoded frames.

[0100] Taking hardware decoding as an example, an embedded device can initiate image processing using the vaBeginPicture() function in the Hardware Acceleration Interface (VAAPI), preparing resources such as the target surface and marking the beginning of a decoding cycle. It then calls vaRenderPicture() in the VAAPI to send the bitstream data required for decoding to the hardware decoder, completing the decoding process. Finally, it calls vaEndPicture() to terminate the image decoding process.

[0101] In some embodiments, the embedded device can perform hardware decoding-related initialization processing, including: obtaining and processing video encoding specifications, querying the hardware decoding layer (VDEC layer) for supported encoding specifications through VAAPI, determining video-related parameters based on the supported encoding specifications, and then determining decoding-related parameters for configuration. The embedded device can complete hardware decoding context creation and hardware decoder initialization through VAAPI.

[0102] Taking software decoding as an example, an embedded device can initialize the software decoder through the avcodec_open2() function, then push the video stream to the software decoder for software decoding through the avcodec_send_packet() function. Next, call avcodec_receive_frame() to obtain the decoded image data frame by frame, that is, obtain the decoded frame.

[0103] After video decoding, the decoded frames can be directly stored in the bound shared buffer so that they can be shared with subsequent modules through the shared buffer. Specifically, if a DMA channel is bound, the embedded device can directly transfer the decoded frames in the shared buffer to the image output module through physical address mapping or buffer reference, without CPU transfer or memory copying.

[0104] For example, the decoded frame is sent to the image output module corresponding to the video decoding task. The image output module includes but is not limited to:

[0105] Image display interface (for local preview or real-time visualization);

[0106] Video re-encoding module (for secondary compression or storage);

[0107] Image analysis engine (for object detection, face recognition, motion analysis, etc.).

[0108] Thanks to the use of DMA or shared memory mapping, decoded frames can be directly accessed at high speed by the image output module, avoiding the high load and latency caused by traditional CPU copying and improving throughput. Furthermore, memory mapping technology reduces the number of data copies, and combined with an intelligent DMA scheduling mechanism, it dynamically adjusts transmission timing and bandwidth allocation to ensure real-time and efficient data transmission.

[0109] Specifically, after the decoded frame of a certain video decoding task is stored in the corresponding shared buffer, the shared buffer can send information to the image output module corresponding to the video decoding task, and then the image output module can directly access the decoded frame from the shared buffer.

[0110] For example, in a multi-tasking concurrent scenario, the system can coordinate access and recycling of different task frames through a frame buffer management mechanism to avoid memory conflicts.

[0111] It should be understood that if the system does not support DMA or zero-copy technology (ie, buffer sharing cannot be achieved), the decoded frame can be sent back to the upper-layer application through a callback mechanism for display or post-processing by the application.

[0112] In some embodiments, after the task is processed, the system reclaims the occupied hardware decoding resources, shared buffers, and DMA channels, and updates the system resource status. The updated system resource status is fed back to the scheduling module to drive the timely scheduling of subsequent tasks.

[0113] For example, you can call the vaDestroyContext() function and the vaDestroySurfaces() function in VAAPI to release hardware resources.

[0114] In some embodiments, during the execution of the video decoding task, an abnormal situation may occur. When the abnormal situation occurs, the embedded device can downgrade the video decoding task according to the importance of the task and the system policy.

[0115] Exemplarily, abnormal situations include but are not limited to: insufficient decoding resources; DMA mapping failure or limited transmission bandwidth; image output module load saturation or overload.

[0116] Exemplarily, the degradation process includes but is not limited to: switching to a software decoding path; reducing image resolution or frame rate; buffering wait; and discarding low-priority tasks to ensure critical path performance.

[0117] For example, if hardware decoding fails or the system load exceeds the limit → fall back to software decoding.

[0118] For example, if the frame rate or resolution is too high and affects system stability, a degradation strategy is triggered to reduce the image frame rate or resolution.

[0119] For example, when resources are extremely tight, low-priority tasks can be discarded to ensure the stable operation of high-priority tasks.

[0120] In some embodiments, if the hardware decoding resources are limited or the task itself does not meet the hardware decoding conditions (such as resolution exceeds the limit), the video decoding task is assigned to the software decoding channel. Exemplarily, whether the video decoding task supports degradation processing is determined by the characteristics of the task. In the case of insufficient hardware decoding resources, the video decoding task that supports degradation processing can be assigned to the software decoding channel, and the software decoding resources can be used for decoding processing. For video decoding tasks that do not support degradation processing, they can be queued in the scheduling queue and wait for the release of hardware decoding resources.

[0121] In some examples, for a first target task, when the resources of the hardware decoding channel are insufficient or decoding using the hardware decoding channel fails, the software decoding channel is switched to the first target task; the first target task includes the first category video decoding task and / or the third category video decoding task.

[0122] In some examples, for a second target task, when the data transmission resources are insufficient, the image resolution or frame rate required by the second target task is reduced, where the second target task includes the second type of video decoding task and / or the third type of video decoding task;

[0123] In some examples, when the load of the image output module is saturated, the video decoding task with a lower priority is cached and waited.

[0124] like Figure 2 As shown, in some embodiments, another video decoding method applicable to an embedded system is provided, which specifically includes the following steps:

[0125] S201: Query the video coding specification supported by the hardware driver layer, and determine whether hardware decoding is supported based on the video coding specification.

[0126] Specifically, the upper-layer application uses the VAAPI interfaces vaQueryConfigEntrypoints and vaQueryConfigProfiles to send query requests to the VDEC layer. The VDEC layer queries the supported encoding formats through the VDEC_IsSupportedFeature interface and returns it to the VAAPI. Based on the VAAPI response, the upper-layer application determines whether hardware decoding is supported.

[0127] S202: If hardware decoding is supported, perform hardware decoding initialization; if hardware decoding is not supported, perform software decoding initialization.

[0128] Specifically, a video decoding context service can be created to initialize the hardware decoder and related interfaces. For example, the VAAPI interface vaCreateContext can be used to initialize the VAAPI layer decoding context decode_context, and the VDEC layer interface VDEC_StreamCreate can be called to initialize the hardware decoder and related interfaces.

[0129] S203. For each video decoding task to be processed, parse the video decoding task to obtain the encoding parameters of the video stream; generate a task feature label corresponding to the video decoding task based on the encoding parameters, the task source information, the task type, and the task requirement information carried in the video decoding task.

[0130] S204 , classifying the priority of each video decoding task to be processed according to the task feature label corresponding to each video decoding task.

[0131] S205 : Detect the decoding resources, data transmission resources, and load information of the image output module in the current system to obtain the current system resource status.

[0132] S206 , according to the current system resource status, preferentially allocating hardware decoding channels to the first type of video decoding tasks; the first type of video decoding tasks are video decoding tasks classified as high-priority real-time tasks.

[0133] S207: Obtain the video stream to be decoded, register a hardware buffer, and bind the hardware buffer to the allocated hardware decoding channel.

[0134] Specifically, the VAAPI interface function vaCreateSurfaces() can be used to call drmIoctl to register the hardware buffer.

[0135] S208 , decoding the video frame. After the hardware decoding layer completes the video decoding task, the decoded frame is sent to the hardware buffer.

[0136] Specifically, after the VDEC layer completes the decoding process, it triggers a completion callback event and returns the decoded frame data to the previously requested hardware buffer.

[0137] S209: Share the hardware buffer to provide the decoded frames in the hardware buffer to the upper layer application.

[0138] Specifically, hardware buffer sharing (shared hardware buffer) is completed through the VAAPI interface function vaExportSurfaceHandle. First, the device file can be obtained through xcb_dri3_open, and secondly, the hardware buffer is shared through the drmPrimeHandleToFD interface to complete the configuration of the DMA shared file descriptor VADRMPRIMESurfaceDescriptor, and then submitted to the upper application by the vaExportSurfaceHandle interface. Exemplarily, the DMA shared file descriptor includes: data type, data format / size, offset, DRM type. The upper application can obtain the decoded frame from the shared hardware buffer based on this configuration information.

[0139] S210 , preferentially allocating software decoding channels to second-category video decoding tasks; the second-category video decoding tasks are video decoding tasks classified as low-priority background tasks.

[0140] S211 . Perform software decoding processing based on the initialized software decoding resources.

[0141] It should be understood that after completing the decoding process of a task, the corresponding hardware decoding resources or software decoding resources can be destroyed. Specifically, the VAAPI interface functions vaDestroyContext and vaDestroySurfaces are used to destroy decoding-related resources. The VDEC layer calls VDEC_Exit to destroy related resources.

[0142] S212: Store the soft-decoded frames into a shared software buffer.

[0143] S213: Provide the decoded frame to an upper-layer application through a shared software buffer.

[0144] It should be understood that after video decoding, some post-processing may be involved. In some embodiments, video post-processing can be accelerated based on VAAPI, including the following steps:

[0145] (1) Query the post-processing specifications supported by the VDEC layer through VAAPI and return them to the upper-layer application. The upper-layer application determines whether hardware post-processing is supported based on the returned supported post-processing specifications. If so, execute step (2). If not, exit the hardware post-processing process.

[0146] Specifically, the VAAPI interfaces vaQueryConfigEntrypoints and vaQueryConfigProfiles can be used to issue query requests to the VDEC layer; the VDEC layer queries the supported encoding formats through the VDEC_IsSupportedFeature interface and returns them to VAAPI. VAAPI further returns the supported post-processing specifications to the upper-layer application, allowing the upper-layer application to make corresponding judgments.

[0147] (2) VAAPI related interfaces perform post-processing hardware related initialization.

[0148] For example, it creates a post-processing context, registers related configurations, and completes the initialization of the hardware interface related to post-processing. Specifically, it uses the VAAPI interface vaCreateContext to initialize the VAAPI layer post-processing context vpcontext and calls the VDEC layer interface VP_StreamCreate to complete the initialization of the hardware post-processing.

[0149] (3) Receive the hardware buffer of the upper-layer application through the DMA interface, complete the hardware buffer registration of the processed frame, and allocate the relevant post-processing parameter configuration of the frame resource through the VAAPI interface.

[0150] For example, the drmprimefdtohandle interface is called through the VAAPI interface vaCreateSurfaces to obtain the shared buffer VADRMPRIMESurfaceDescriptor and complete the registration of the output frame.

[0151] (4) Post-processing is performed on the pending frame through the VAAPI interface. After the hardware layer completes the post-processing process, the processed frame data is sent to the hardware buffer.

[0152] (5) Sharing the hardware buffer that has completed processing and sharing the processed frames to the upper-layer application through the VAAPI interface.

[0153] The VAAPI interface function vaExportSurfaceHandle is used to share the hardware buffer for output frames. First, the device file is obtained through xcb_dri3_open, followed by the hardware buffer sharing interface drmPrimeHandleToFD. The DMA shared file descriptor VADRMPRIMESurfaceDescriptor is configured and then submitted to the upper-layer application through the vaExportSurfaceHandle interface.

[0154] (6) Determine whether all video frames are post-processed. If so, proceed to step (7). Otherwise, return to step (3) to perform post-processing on the next frame.

[0155] (7) Destroy the post-processing context and release VAAPI resources.

[0156] Specifically, the VAAPI interface functions vaDestroyContext and vaDestroySurfaces are used to destroy the post-processing related hardware resources. The VDEC layer calls VDEC_Exit to destroy the related resources.

[0157] This solution, combined with the hardware driver layer and the VAAPI video processing specification, implements a series of post-processing interfaces, accepts video frame post-processing tasks from upper-layer applications, and completes a highly efficient, zero-copy, high-speed display delivery process. This enables VAAPI-compatible video applications on embedded devices to achieve faster display response times, freeing up CPU resources, improving overall system efficiency, and reducing power consumption.

[0158] It should be understood that although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0159] Based on the same inventive concept, embodiments of the present application further provide a video decoding device for embedded systems, for implementing the methods described in the aforementioned embodiments. The implementation solutions provided by this device are similar to the implementation solutions described in the aforementioned methods. Therefore, the specific limitations in one or more embodiments of the video decoding device for embedded systems provided below can be found in the aforementioned limitations on the video decoding method for embedded systems, and will not be further elaborated here.

[0160] like Figure 3 As shown, in one embodiment, a video decoding device suitable for an embedded system is provided, the device comprising:

[0161] The task parsing module 31 is used to parse each acquired video decoding task to be processed and obtain encoding parameters of the video stream;

[0162] A task feature generation module 32 is configured to generate a task feature label corresponding to the video decoding task based on the encoding parameters, the task source information, the task type, and the task requirement information carried in the video decoding task;

[0163] A priority classification module 33 is used to classify the priorities of the video decoding tasks to be processed according to the task feature label corresponding to each video decoding task;

[0164] The resource status detection module 34 is used to detect the decoding resources, data transmission resources and image output module load information in the current system to obtain the current system resource status; the image output module is the module to which the decoded frame is output after decoding;

[0165] The decoding channel allocation module 35 is used to give priority to allocating hardware decoding channels for the first type of video decoding tasks and give priority to allocating software decoding channels for the second type of video decoding tasks according to the current system resource status; wherein the first type of video decoding tasks are video decoding tasks classified as high-priority real-time tasks, and the second type of video decoding tasks are video decoding tasks classified as low-priority background tasks.

[0166] In some embodiments, the decoding resources include the number of idle channels of the hardware decoding unit, the encoding formats supported by the hardware decoding unit, and the schedulable computing resources of the software decoding module.

[0167] In some embodiments, the data transmission resources include an idle direct memory access channel, a data sharing mechanism with the image output module, and a current transmission bandwidth.

[0168] In some embodiments, the load information of the image output module includes information representing the current processing capability of the image output module.

[0169] In some embodiments, the task feature tag includes at least one of a real-time level tag, a fault tolerance level, degradability, a time priority level, a resource dependency strength, a flow feature tag, or a processing chain dependency.

[0170] In some embodiments, the real-time level label includes high real-time, medium real-time or low real-time; the fault tolerance level includes strict, medium or loose; the degradability is used to characterize whether it supports reduced resolution and / or frame rate processing; the high or low time priority level is positively correlated with the strength of time sensitivity; the resource dependence strength is used to characterize whether it is strongly dependent on hardware decoding resources, and / or whether shared buffer resources are required; the stream feature label includes long continuous, short burst, high bandwidth or low bit rate; the processing chain dependency is used to characterize whether it needs to flow into the image output module immediately after decoding.

[0171] like Figure 4 As shown, in some embodiments, the device further includes:

[0172] The sharing module 36 is used to bind the corresponding shared buffer to the hardware decoding channel and / or the software decoding channel; after video decoding, the decoded frame is stored in the bound shared buffer; the decoded frame refers to the image frame after decoding the video stream; the decoded frame in the shared buffer is directly transmitted to the image output module through physical address mapping or buffer reference.

[0173] In some embodiments, the classification category obtained also includes medium-priority quasi-real-time tasks; the real-time requirement of the medium-priority quasi-real-time tasks is lower than that of the high-priority real-time tasks, but higher than that of the low-priority background tasks; the video decoding tasks classified as the medium-priority quasi-real-time tasks are third-category video decoding tasks.

[0174] In some embodiments, the decoding channel allocation module 35 is further configured to preferentially allocate the hardware decoding channel to the third type of video decoding task if there are still idle hardware decoding channels after the hardware decoding channels are allocated to the first type of video decoding task.

[0175] In some embodiments, the decoding channel allocation module 35 is also used to switch and allocate the software decoding channel to the first target task for the first target task when the resources of the hardware decoding channel are insufficient or decoding using the hardware decoding channel fails; the first target task includes the first category video decoding task and / or the third category video decoding task.

[0176] In some embodiments, the decoding channel allocation module 35 is also used to reduce the image resolution or frame rate required by the second target task for the second target task when the data transmission resources are insufficient. The second target task includes the second type of video decoding task and / or the third type of video decoding task.

[0177] In some embodiments, the decoding channel allocation module 35 is further configured to cache the video decoding tasks with lower priorities when the load of the image output module is saturated.

[0178] In some embodiments, the resource status detection module 34 is further configured to call a hardware acceleration interface to detect the load information of decoding resources, data transmission resources, and image output modules in the current system to obtain the current system resource status.

[0179] In some embodiments, an embedded device is provided, whose internal structure diagram can be as follows: Figure 5 As shown. The embedded device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the embedded device is used to provide computing and control capabilities. The memory of the embedded device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the embedded device is used to exchange information between the processor and an external device. The communication interface of the embedded device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video decoding method suitable for an embedded system is implemented.

[0180] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the embedded device to which the solution of the present application is applied. The specific embedded device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0181] In one embodiment, an embedded device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in each embodiment of the present application when executing the computer program.

[0182] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in each embodiment of the present application are implemented.

[0183] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in each embodiment of the present application when executed by a processor.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0185] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0186] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0187] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A video decoding method suitable for embedded systems, characterized in that: The method comprises: For each acquired video decoding task to be processed, the video decoding task is parsed to obtain encoding parameters of the video stream; Generate a task feature label corresponding to the video decoding task according to the encoding parameters, the task source information, the task type, and the task requirement information carried in the video decoding task; Prioritizing each of the video decoding tasks to be processed according to the task feature label corresponding to each of the video decoding tasks; Detecting the decoding resources, data transmission resources, and load information of the image output module in the current system to obtain the current system resource status; the image output module is the module to which the decoded frame is output after decoding; According to the current system resource status, hardware decoding channels are preferentially allocated to the first type of video decoding tasks, and software decoding channels are preferentially allocated to the second type of video decoding tasks; wherein, the first type of video decoding tasks are video decoding tasks classified as high-priority real-time tasks, and the second type of video decoding tasks are video decoding tasks classified as low-priority background tasks.

2. The method according to claim 1, characterized in that The decoding resources include the number of idle channels of the hardware decoding unit, the encoding formats supported by the hardware decoding unit, and the schedulable computing resources of the software decoding module; The data transmission resources include an idle direct memory access channel, a data sharing mechanism with the image output module, and a current transmission bandwidth; The load information of the image output module includes information representing the current processing capability of the image output module.

3. The method according to claim 1, characterized in that The task feature tag includes at least one of a real-time level tag, a fault tolerance level, degradability, a time priority level, a resource dependency strength, a flow feature tag, or a processing chain dependency; The real-time level label includes high real-time, medium real-time or low real-time; Said tolerance levels include strict, moderate or loose; The degradability is used to indicate whether resolution and / or frame rate reduction processing is supported; The level of the time priority is positively correlated with the strength of the time sensitivity; The resource dependency strength is used to indicate whether the system is strongly dependent on hardware decoding resources and / or whether shared buffer resources are required; The flow feature tags include long-term continuous, short-term burst, high bandwidth or low bit rate; The processing chain dependency is used to indicate whether the decoded data needs to be immediately fed into the image output module.

4. The method according to claim 1, wherein The method further comprises: Binding a corresponding shared buffer to the hardware decoding channel and / or the software decoding channel; After the video is decoded, the decoded frame is stored in the bound shared buffer; the decoded frame refers to the image frame after the video stream is decoded; The decoded frame in the shared buffer is directly transmitted to the image output module through physical address mapping or buffer reference.

5. The method according to claim 1, wherein The obtained classification category further includes medium-priority quasi-real-time tasks; the real-time requirement of the medium-priority quasi-real-time tasks is lower than that of the high-priority real-time tasks, but higher than that of the low-priority background tasks; The video decoding task classified as the medium-priority quasi-real-time task is a third-category video decoding task; The method further comprises: After hardware decoding channels are allocated to the first type of video decoding tasks, if idle hardware decoding channels still exist, the hardware decoding channels are preferentially allocated to the third type of video decoding tasks.

6. The method according to claim 1, wherein The method further comprises: For a first target task, if resources of the hardware decoding channel are insufficient or decoding using the hardware decoding channel fails, switching allocation of the software decoding channel to the first target task; the first target task includes the first type of video decoding task and / or the third type of video decoding task; and / or, For the second target task, when the data transmission resources are insufficient, reducing the image resolution or frame rate required by the second target task, the second target task including the second type of video decoding task and / or the third type of video decoding task; and / or, When the load of the image output module is saturated, the video decoding task with a lower priority is cached and waits.

7. The method according to any one of claims 1 to 6, characterized in that The detecting of the load information of the decoding resources, data transmission resources and image output module in the current system to obtain the current system resource status includes: Call the hardware acceleration interface to detect the decoding resources, data transmission resources and image output module load information in the current system to obtain the current system resource status.

8. A video decoding device suitable for an embedded system, characterized in that: The device comprises: A task parsing module is used to parse each acquired video decoding task to be processed and obtain encoding parameters of the video stream; A task feature generation module is used to generate a task feature label corresponding to the video decoding task based on the encoding parameters, the task source information, the task type and the task requirement information carried in the video decoding task; A priority classification module, configured to classify the priorities of the video decoding tasks to be processed according to the task feature labels corresponding to the video decoding tasks; A resource status detection module is used to detect the load information of the decoding resources, data transmission resources and image output module in the current system to obtain the current system resource status; the image output module is the module to which the decoded frame is output after decoding; A decoding channel allocation module is used to prioritize the allocation of hardware decoding channels for the first category of video decoding tasks and prioritize the allocation of software decoding channels for the second category of video decoding tasks based on the current system resource status; wherein the first category of video decoding tasks is a video decoding task classified as a high-priority real-time task, and the second category of video decoding tasks is a video decoding task classified as a low-priority background task.

9. An embedded device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images