A video decoding method, device, electronic equipment and medium

Abstract

The present disclosure provides a video decoding method, device, electronic equipment and medium, relates to the technical field of video transmission, and particularly relates to the technical field of video decoding. A specific implementation scheme is as follows: a decoding request periodically sent by a video processing module is received. A to-be-decoded video frame is parsed from each received decoding request, and a hardware decoder is called to decode the to-be-decoded video frame. Furthermore, a decoded video frame that has been completed decoding is asynchronously acquired, and the decoded video frame that has been completed decoding is returned to the video processing module. Thus, the speed of video decoding is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video transmission, and particularly relates to the technical field of video decoding. BACKGROUND

[0002] With the increasing richness of multimedia data, video has become an important data form in people's communication and learning, etc. Since the data volume of video is usually large, in order to guarantee the transmission speed and video quality, the video usually needs to be encoded and decoded for transmission. SUMMARY

[0003] The present disclosure provides a video decoding method, device, electronic equipment and medium.

[0004] In a first aspect, the present disclosure provides a video decoding method, comprising:

[0005] receiving a decoding request periodically sent by a video processing module;

[0006] parsing a to-be-decoded video frame from the decoding request, and calling a hardware decoder to decode the to-be-decoded video frame, every time a decoding request is received;

[0007] asynchronously obtaining a decoded video frame that has been completed decoding, and returning the decoded video frame that has been completed decoding to the video processing module.

[0008] In a second aspect, the present disclosure provides a video decoding device, comprising:

[0009] a receiving module, configured to receive a decoding request periodically sent by a video processing module;

[0010] a calling module, configured to parse a to-be-decoded video frame from the decoding request, and call a hardware decoder to decode the to-be-decoded video frame, every time a decoding request is received;

[0011] a returning module, configured to asynchronously obtain a decoded video frame that has been completed decoding, and return the decoded video frame that has been completed decoding to the video processing module.

[0012] In a third aspect, the present disclosure provides an electronic equipment, comprising:

[0013] at least one processor; and

[0014] a memory in communication with the at least one processor; wherein

[0015] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method in any one of the first aspect.

[0016] In a fourth aspect, the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method according to any one of the first aspect.

[0017] In a fifth aspect, the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of the first aspect.

[0018] It should be understood that the description of the contents of this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0020] Figure 1 is a flowchart of a video decoding method provided by an embodiment of the present disclosure;

[0021] Figure 2 is a flowchart of another video decoding method provided by an embodiment of the present disclosure;

[0022] Figure 3 is an exemplary schematic diagram of a video decoding process provided by an embodiment of the present disclosure;

[0023] Figure 4 is a structural schematic diagram of a video decoding apparatus provided by an embodiment of the present disclosure;

[0024] Figure 5 is a block diagram of an electronic device for implementing the video decoding method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0026] Due to the large amount of data of a video and the complex data structure, the efficiency of a tool for processing the video is required to be high. Fast Forward Mpeg (FFmpeg) is one of the most widely used audio and video processing tools, which can realize video encoding and decoding, streaming, transcoding and other functions, and can be applied to short video or live broadcast and other practical application scenarios, and has high reliability and security.

[0027] When decoding a video in FFmpeg, a software decoding method is generally used, but the software decoding depends on a central processing unit (CPU), and the CPU also needs to undertake other work, resulting in slow software decoding speed.

[0028] To improve the video decoding speed, an embodiment of the present disclosure provides a video decoding method, which can be applied to an electronic device and specifically can be applied to a video processing framework running in the electronic device. For example, the electronic device can be a server, a desktop computer or a notebook computer and the like with video processing capability, and the video processing framework in the electronic device can be FFmpeg or Gstreamer and the like. As shown in the figure, the method comprises the following steps: Figure 1

[0029] S101, receiving a decoding request periodically sent by a video processing module.

[0030] The video processing module is a logical module in the video processing framework, i.e. a software module, which is used to process the video decoding process. For convenience of description, the software module for implementing the video decoding method provided by the embodiment of the present disclosure is referred to as a video decoding module.

[0031] When the video processing module needs to decode a to-be-decoded video frame in a to-be-decoded video, a first interface of the video decoding module is called, so that a decoding request is sent to the video decoding module once. At this time, the decoding request includes an audio video packet (AVPacket) structure, and the AVPacket includes a to-be-decoded video frame and metadata of the to-be-decoded video frame. For example, the metadata includes the size, format and alignment information of the video frame. The size of the video frame includes width and height, for example, the size of the video frame, i.e. width x height, can be 4096; the format of the video frame can be color encoding method 420 (YUV420) and the like.

[0032] S102, parsing a to-be-decoded video frame from each received decoding request, and decoding the to-be-decoded video frame by calling a hardware decoder.

[0033] ​The video decoding module parses a to-be-decoded video frame from an AVPacket included in the decoding request each time a decoding request is received, and calls the hardware decoder to decode the to-be-decoded video frame.

[0034] The hardware decoder in the embodiments of the present disclosure can be an Ascend series neural-network processing unit (NPU) or a Digital Signal Processing (DSP) hardware decoder, and the embodiments of the present disclosure do not make specific limitations thereto.

[0035] The Ascend NPU is a high-performance computing chip specially designed for the field of Artificial Intelligence (AI) data computing, is a main high-performance chip produced by a mainstream AI chip manufacturer, and has a running capability as high as tens of trillions of operations per second (TOPS), so the application range is very wide. The Ascend NPU has a powerful Digital Video Pre-Processing (DVPP) module, and using the Ascend NPU for video hardware decoding in FFmpeg can greatly improve the video decoding speed of FFmpeg. Moreover, compared with the software decoding mode, the hardware decoding also has the advantage of low power consumption.

[0036] S103, asynchronously obtaining a decoded video frame that has been completed decoding, and returning the decoded video frame that has been completed decoding to the video processing module.

[0037] In the embodiments of the present disclosure, S102 and S103 are executed asynchronously, that is, each time the video decoding module receives a decoding request, the hardware decoder is called to decode a to-be-decoded video frame in the decoding request, and meanwhile the video decoding module also obtains and returns a decoded video frame that has been completed decoding to the video processing module.

[0038] It can be seen that the decoding process and the returning process of the video frame are asynchronous, so that the next video frame is not decoded until the decoding of the current video frame is completed and the current video frame is returned, thereby improving the video decoding efficiency.

[0039] Through the above method, the embodiments of the present disclosure can call the hardware decoder in the video processing framework, and use the hardware decoder to decode a to-be-decoded video frame, that is, use the hardware decoding mode to decode the video. The hardware decoding does not depend on the CPU, so compared with the software decoding mode, the decoding speed is faster, and therefore the embodiments of the present disclosure can improve the video decoding speed.

[0040] Currently, since the Ascend NPU cannot support FFmpeg, and there is no integration solution for the Ascend NPU in FFmpeg, the developer cannot directly call the Ascend NPU for video decoding through the interface or function in FFmpeg.

[0041] The embodiment of the present disclosure realizes calling the Ascend NPU for video decoding in FFmpeg, so that the developer does not need to perform additional development work for FFmpeg and the Ascend NPU when using the Ascend NPU in FFmpeg, and therefore does not need to be familiar with both FFmpeg and the Ascend NPU, thereby reducing the requirements on the developer, and also reducing the developer level, and reducing the influence on the stability, reliability and efficiency of the communication between FFmpeg and the Ascend NPU.

[0042] The video decoding method provided by the embodiment of the present disclosure is described in detail as follows:

[0043] In the embodiment of the present disclosure, the video decoding module can include two software modules, a first decoding module and a second decoding module. The first decoding module can also be referred to as a decoder (ff_atlas_decoder) module interacting with FFmpeg, and the second decoding module can also be referred to as a decoding module.

[0044] The first decoding module includes a structure definition unit, an initialization unit, a decoding unit, a discarding unit and a closing unit. The initialization unit can also be referred to as an init_decoder unit, the decoding unit can also be referred to as a decode unit, the discarding unit can also be referred to as a flush unit, and the closing unit can also be referred to as a close_decoder unit.

[0045] The structure definition unit defines the type of the video decoding module, whether to support flushing, whether to support multi-threading and other capabilities. Moreover, the structure definition unit also includes a filter for converting the encapsulation format of the to-be-decoded video frame. For example, the filter can convert the encapsulation format of the to-be-decoded video frame from Moving Picture Experts Group 4 (MP4) to Annex-B format.

[0046] The init_decoder unit is configured to perform an initialization process, the decode unit is configured to perform a decoding process of the to-be-decoded video frame and a returning process of the decoded video frame, the flush unit is configured to perform a video frame discarding process, and the close_decoder unit is configured to perform a closing process of the hardware decoder. Details of the processes can be referred to subsequent descriptions.

[0047] Before decoding the to-be-decoded video frame, that is, before S101, an initialization process is performed first, which includes the following steps.

[0048] Step 1: receiving the code stream basic information of the to-be-decoded video sent by the video processing module.

[0049] The code stream basic information includes video frame metadata such as the width and height of the to-be-decoded video frame, and code stream metadata such as the code stream encoding type and the code stream specification. For example, the code stream encoding type can be a digital video compression format (H.264) or a video encoding standard (H.265), etc. The code stream specification can be a low complexity (Low Complexity) specification or a high efficiency (High Efficiency) specification, etc.

[0050] Optionally, after receiving the code stream basic information sent by the video processing module, the first decoding module can also verify the code stream basic information. For example, verifying whether the video frame size obtained by multiplying the width of the to-be-decoded video frame by the height is less than or equal to a preset maximum size, and verifying whether the code stream encoding type belongs to a specified encoding type set, etc. And after the verification is passed, step 2 is performed.

[0051] Step 2: initializing itself and the hardware decoder based on the code stream basic information.

[0052] When the video decoding module is software initialized, the init_decoder unit of the first decoding module sends the code stream basic information to the second decoding module, the second decoding module verifies the code stream basic information, creates a result cache queue after the verification is passed, and locks the result cache queue, thereby improving the security of the decoded video frame. Then the second decoding module generates a hardware decoding channel according to the video frame metadata, wherein the hardware decoding channel is used to record decoding information such as the video frame metadata. Moreover, the second decoding module also starts an asynchronous decoding video frame callback thread, so that the subsequent hardware decoder can use the callback thread to call back the callback unit of the second decoding module, and the callback unit obtains the decoded video frame completed by the hardware decoder.

[0053] After the init decoder unit of the first decoding module initializes the second decoding module, an internal decoding object is generated, which records the handle of the second decoding module, interface information of the second decoding module, and transmission channel information between the first decoding module and the second decoding module, and the like. The handle is used for communication with the second decoding module.

[0054] The video decoding module can also perform hardware initialization on the hardware decoder, including checking hardware device information of the hardware decoder, configuring a context environment of the hardware decoder, and other basic configuration items. The init decoder unit of the first decoding module can directly initialize the hardware decoder, or the init decoder unit can initialize the hardware decoder through the second decoding module.

[0055] Through the above method, the video decoding module in the embodiment of the disclosure can initialize itself and the hardware decoder, thereby completing the preparation work before decoding, and facilitating the subsequent video frame decoding process.

[0056] After receiving each decoding request in S102, the video decoding module can parse the metadata of the to-be-decoded video frame from the decoding request through the decode unit of the first decoding module, and check the metadata. For example, the video frame size in the metadata can be checked to be the video frame size carried by the basic information of the code stream during initialization, and the video frame format in the metadata can be checked to be the video frame format carried by the basic information of the code stream during initialization. After the check passes, the decode unit organizes the to-be-decoded video frame and the metadata into a preset structure and stores it in the host memory, and calls the second decoding module through the handle of the second decoding module saved in advance, thereby sending the memory address of the structure to the second decoding module.

[0057] Then, the second decoding module stores the to-be-decoded video frame into the memory of the hardware decoder, and sends the metadata of the to-be-decoded video frame to the hardware decoder, thereby calling the hardware decoder to decode the to-be-decoded video frame.

[0058] The hardware decoder manages an asynchronous to-be-decoded queue, which includes to-be-decoded video frames and their metadata that have not been completed decoding. The hardware decoder obtains the to-be-decoded video frame from the to-be-decoded queue and decodes it to obtain a decoded video frame, and stores the decoded video frame and its metadata in the memory of the hardware decoder.

[0059] In the embodiment of the disclosure, the video decoding module can first check the metadata of the to-be-decoded video frame before calling the hardware decoder to decode the video frame, which can improve the security of the decoding process, reduce the situation of sending to-be-decoded video frames that the hardware decoder cannot decode to the hardware decoder, and improve the decoding efficiency.

[0060] After the video decoding module invokes the hardware decoder in S102, the video decoding module can further obtain the decoded video frame from the hardware decoder. Figure 2 The process of obtaining the decoded video frame from the hardware decoder by the video decoding module includes the following steps:

[0061] S201, when being called back by the hardware decoder, obtaining the decoded video frame from the hardware decoder memory.

[0062] Whenever the hardware decoder completes decoding a decoded video frame, the second decoding module of the video decoding module is called back, so that the second decoding module copies the decoded video frame in the hardware decoder memory to the host memory.

[0063] S202, packaging the decoded video frame into a specified format, and storing the decoded video frame in the specified format in the result cache queue.

[0064] The specified format is a format supported by the video processing module. For example, the specified format is an audio video frame (AVFrame) format.

[0065] The second decoding module can package the decoded video frame and its metadata into the specified format and store them in the result cache queue.

[0066] Meanwhile, the second decoding module can also delete the to-be-decoded video frame and its metadata stored in the hardware decoder this time, that is, delete the data corresponding to the decoded video frame and its metadata obtained this time before decoding, so as to avoid memory leakage and ensure the safety of the decoding process.

[0067] On this basis, the manner of obtaining the decoded video frame in S103 can be implemented as follows: obtaining the decoded video frame in the specified format from the result cache queue.

[0068] Specifically, when receiving a decoding request, the decode unit of the first decoding module can invoke the second decoding module, so that the second decoding module obtains the decoded video frame in the specified format from the result cache queue and returns the obtained decoded video frame to the decode unit.

[0069] In the embodiment of the present disclosure, the video frame decoding process and the returning process are asynchronous, so when receiving a decoding request, the decode unit stores the to-be-decoded video frame and its metadata in the host memory, and when invoking the second decoding module, the decode unit also sends the memory address of the storage to the second decoding module. When the second decoding module is invoked by the decode unit, the second decoding module also asynchronously invokes the hardware decoder to decode the to-be-decoded video frame.

[0070] Through the method, the video decoding module can obtain the decoded video frame in time when the hardware decoder finishes decoding a video frame in the embodiments of the present disclosure, so that the decoded video frame can be returned in time when the next decoding request is received, and the return efficiency of the decoded video frame is ensured.

[0071] In the embodiments of the present disclosure, the hardware decoder needs a certain time to decode a video frame, and there are some cases where a video frame can be decoded only by relying on a previous video frame and / or a subsequent video frame. Therefore, the video decoding module may not obtain the decoded video frame after receiving the decoding request.

[0072] Therefore, the video decoding module can notify the video processing module that there is no decoded video frame after receiving the decoding request in S102.

[0073] For example, the decode unit of the video decoding module can send a data packet in the AVFrame format to the video processing module, the data packet does not contain a decoded video frame, and a flag bit indicates that there is no decoded video frame.

[0074] Through the method, the video decoding module can timely notify the video processing module when there is no decoded video frame, so that the video processing module can avoid mistakenly considering that the video decoding module does not respond, and the normal operation of the video decoding process is ensured.

[0075] In the embodiments of the present disclosure, the hardware decoder will call back the video decoding module as long as the decoding of a to-be-decoded video frame is finished, regardless of whether the decoding is successful. Therefore, in order to avoid obtaining a decoding failure result, the video decoding module can further perform the following steps before obtaining the decoded video frame from the memory of the hardware decoder: when being called back by the hardware decoder, determining whether the result flag bit of the callback parameter indicates that the decoding is successful.

[0076] If yes, the step of obtaining the decoded video frame from the memory of the hardware decoder in S201 is performed.

[0077] If no, the decoded video frame is not obtained from the hardware decoder, and the next callback of the hardware decoder is waited.

[0078] Specifically, when the callback unit of the second decoding module is called back by the hardware decoder, it is determined whether the result flag bit of the callback parameter indicates that the decoding is successful. If yes, the callback unit obtains the decoded video frame from the memory of the hardware decoder. If no, the callback unit waits for the next callback of the hardware decoder.

[0079] Since the decoding failure result obtained when the hardware decoder fails to decode is useless for the video processing module and can also affect subsequent video playing and the like, only the decoded video frames with successful decoding can be obtained.

[0080] In the embodiment of the present disclosure, when the video processing module sends the last to-be-decoded video frame of the to-be-decoded video, at this time, there is no to-be-decoded video frame, the video processing module can inform the video decoding module through the specified flag bit of the decoding request. That is, in the decoding request sent by the video processing module and including the last to-be-decoded video frame, the specified flag bit indicates that the to-be-decoded video frame has been sent and ended.

[0081] Therefore, the video decoding module can also identify the specified flag bit included in each decoding request after receiving the decoding request. When the specified flag bit is identified for the first time and indicates that the to-be-decoded video frame has been sent and ended, the hardware decoder is informed to reset after decoding the to-be-decoded video frame that is currently not decoded.

[0082] The decode unit of the first decoding module identifies, for the first time, that the specified flag bit in the AVPacket in the decoding request is a preset value, which indicates that the to-be-decoded video frame has been sent and ended. At this time, the decode unit sends an end of sequence (eos) mark to the second decoding module, where the eos mark indicates that the to-be-decoded video frame has been sent and ended. After receiving the eos mark, the second decoding module sends an eos' mark to the hardware decoder, where the eos' mark indicates that the to-be-decoded video frame has been sent and ended. Thus, the hardware decoder is caused to reset after decoding the to-be-decoded video frame that is currently not decoded. The eos mark and the eos' mark are only used for distinguishing in the description and are not used to limit the marks themselves. In actual application, the eos mark and the eos' mark can be the same or different.

[0083] Meanwhile, the second decoding module can also inform and wait for the callback thread to exit, and cancel the memory resource applied for the hardware decoder at initialization and destroy the result cache queue to release the space and reduce the impact on the next decoding.

[0084] Through the above method, in the embodiment of the present disclosure, the video decoding module can timely inform the hardware decoder to reset after decoding, thereby improving the efficiency of the hardware decoder reset and reducing unnecessary occupation of the hardware decoder.

[0085] In the embodiments of the present disclosure, since the video processing module cannot obtain decoded video frames every time the decode request is sent, the video processing module can continue to call the first interface of the video decoding module, i.e., continue to send the decode request, to obtain the decoded video frames that have not been returned after sending all the to-be-decoded video frames of the to-be-decoded video. At this time, the video processing module does not include the to-be-decoded video frame in the other decode requests sent after sending the last to-be-decoded video frame, and the specified flag bit in the other decode requests indicates that the to-be-decoded video frame has been sent.

[0086] After the video processing module resets the hardware decoder after the hardware decoder completes decoding of the to-be-decoded video frame that has not been decoded, the video processing module can further perform: if the decode request is received for a specified number of times continuously and the decoded video frame that has been decoded is not obtained, the video processing module is notified that the decoding is completed, so that the video processing module stops sending the decode request.

[0087] After the video processing module resets the hardware decoder after the hardware decoder completes decoding of the to-be-decoded video frame that has not been decoded, the video processing module can further perform: if the decode request is received for a specified number of times continuously and the decoded video frame that has been decoded is not obtained, the video processing module is notified that the decoding is completed, so that the video processing module stops sending the decode request.

[0088] Through the above method, after the video processing module sends all the to-be-decoded video frames, the video processing module can continue to obtain the decoded video frames that have not been returned by sending the decode request that does not include the to-be-decoded video frame. At the same time, the video decoding module can notify the video processing module that the decoding is completed after all the decoded video frames are returned, so as to avoid the video processing module from continuously sending the decode request after the decoding is completed, and to ensure the timeliness of the processing state transmission between the video processing module and the hardware decoder.

[0089] In the video viewing scenario, there can be a video jump scenario. After the video jumps, the video frames close to the currently played video frame lose the playing demand, and the video frames to which the video jumps need to be played. In this scenario, the video decoding module can further perform: receiving the discard notification sent by the video processing module, and discarding the decoded video frame that has been decoded. The discard notification is used to indicate that the decoded video frame that has been decoded is discarded.

[0090] The video processing module can call the second interface of the first decoding module, so as to send the discard notification to the first decoding module. At this time, the flush unit in the first decoding module takes all the decoded video frames from the result cache queue and discards them. Alternatively, the flush unit can notify the second decoding module to take all the decoded video frames from the result cache queue and discard them.

[0091] When the video jumps, the currently decoded video frames have lost the demand for playing, and there is no need to return these decoded video frames to the video processing module, so these decoded video frames can be discarded, reducing the occupation of useless decoded video frames to the memory.

[0092] When it is not necessary to continue to use the hardware decoder for decoding, for example, manually selecting to switch to software decoding, or closing the video in advance during the video playing process, the video processing module can notify the video decoding module to end decoding, at which time the video decoding module can perform the following steps:

[0093] Step 1, receiving the decoding end notification sent by the video processing module. The decoding end notification is used to indicate the end of decoding.

[0094] The video processing module can call the third interface of the first decoding module, so as to send the decoding end notification to the first decoding module.

[0095] Step 2, determining whether the hardware decoder has been notified to reset after decoding the currently undecoded to-be-decoded video frame. If not, step 3 is performed; if yes, no further notification is performed.

[0096] After the first decoding module sends the eos flag to the second decoding module, it can record the current state as having notified the hardware decoder to reset after decoding the currently undecoded to-be-decoded video frame. So that when step 2 is performed, the close_decoder unit of the first decoding module can determine whether the hardware decoder has been notified to reset after decoding the currently undecoded to-be-decoded video frame by checking the current state.

[0097] Step 3, notifying the hardware decoder to reset after decoding the currently undecoded to-be-decoded video frame.

[0098] The method for the close_decoder unit of the first decoding module to notify the hardware decoder to reset after decoding the currently undecoded to-be-decoded video frame is the same as the method for the decode unit to notify the hardware decoder, which can be referred to the above description, and will not be repeated here.

[0099] Through the above method, in the embodiments of the present disclosure, the video decoding module can timely inform the hardware decoder when the hardware decoder is not needed to continue decoding, so that the hardware decoder can be reset in time, reduce unnecessary occupation of the hardware decoder, and improve the timeliness of processing demand transmission between the video processing module and the hardware decoder.

[0100] Moreover, since the embodiments of the present disclosure can implement decoding using a hardware decoder in a video processing framework, the embodiments of the present disclosure can flexibly select or switch a hardware decoding mode or a software decoding mode during video decoding, thereby enriching the ecological environment of video processing schemes of the video processing framework, and making the application range of the video processing framework and the hardware decoder wider.

[0101] Referring to Figure 3 The specific process of video decoding provided by the embodiments of the present disclosure will be described below in combination with actual application scenarios:

[0102] The video decoding module includes an ff_atlas_decoder module and a decoding module, and the video decoding module and the video processing module are both in an FFmpeg framework.

[0103] First, the video processing module calls an initialization interface of the ff_atlas_decoder module, so as to send basic information of a code stream of a video to be decoded to the ff_atlas_decoder module. An init_decoder unit of the ff_atlas_decoder module performs software initialization on the decoding module based on the basic information of the code stream, and performs hardware initialization on an Ascend NPU through the decoding module based on the basic information of the code stream.

[0104] Then, the video processing module periodically and continuously calls a first interface of the ff_atlas_decoder module, so as to continuously send a decoding request to the ff_atlas_decoder module.

[0105] A filter (Filter) of the ff_atlas_decoder module converts an encapsulation format of data carried in the decoding request into an encapsulation format supported by the Ascend NPU. Then, a decode unit of the ff_atlas_decoder module parses a video frame to be decoded and metadata from an AVPacket included in the decoding request, organizes the video frame to be decoded and the metadata into a specified format, stores the video frame to be decoded and the metadata into a memory, and sends a storage address of the memory to the decoding module.

[0106] Meanwhile, the decode unit also asynchronously obtains the data packet in the AVFrame format from the result cache queue through the decoding module, wherein the data packet includes the decoded video frame and the metadata thereof, and then returns the data packet in the AVFrame format to the video processing module.

[0107] Correspondingly, when the decoding (vdec) unit of the decoding module receives the memory address sent by the decode unit, the decoding (vdec) unit stores the to-be-decoded video frame into the memory of the Ascend NPU according to the memory address, and sends the metadata to the Ascend NPU. Meanwhile, the decoding module also asynchronously obtains the data packet in the AVFrame format from the result cache queue, and returns the data packet to the decode unit.

[0108] The Ascend NPU stores the to-be-decoded video frame and the metadata thereof into the decoding queue through the DVPP decoding module, and obtains the to-be-decoded video frame from the decoding queue for decoding. After decoding a decoded video frame, the DVPP decoding module calls back the decoding module, so that the callback unit of the decoding module copies the decoded video frame and the metadata thereof from the memory of the Ascend NPU to the host memory, organizes the decoded video frame and the metadata thereof into a data packet in the AVFrame format, and stores the data packet in the result cache queue.

[0109] When the video jump occurs, the video processing module sends a discard notification to the ff_atlas_decoder module, and the flush unit of the ff_atlas_decoder module notifies the decoding module to take out all the data packets in the AVFrame format from the result cache queue and discard them.

[0110] When the use of the Ascend NPU for video decoding ends, the video processing module sends an end-of-decoding notification to the ff_atlas_decoder module. The close_decoder unit of the ff_atlas_decoder module determines whether the eos flag has been sent to the decoding module, and if not, sends the eos flag to the decoding module.

[0111] When the decoding module receives the eos flag, the decoding module sends an eos' flag to the Ascend NPU, so that the Ascend NPU resets after decoding the to-be-decoded video frame that is currently not decoded.

[0112] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of the video information involved all comply with the relevant legal regulations and do not violate public order and good customs.

[0113] Based on the same inventive concept, corresponding to the above method embodiments, this disclosure also provides a video decoding device, such as... Figure 4 As shown, the device includes: a receiving module 401, a calling module 402, and a returning module 403;

[0114] The receiving module 401 is used to receive decoding requests periodically sent by the video processing module;

[0115] Module 402 is called to parse the video frame to be decoded from each decoding request received and call the hardware decoder to decode the video frame to be decoded.

[0116] Return module 403 is used to asynchronously obtain a decoded video frame that has been decoded and return it to the video processing module.

[0117] In some embodiments of this disclosure, the device may further include:

[0118] The acquisition module is used to retrieve decoded video frames that have been decoded from the hardware decoder memory when the hardware decoder calls back.

[0119] The storage module is used to encapsulate the decoded video frames that have been decoded into a specified format and store the decoded video frames in the specified format into the result cache queue. The specified format is a format supported by the video processing module.

[0120] Return module 403, specifically used for:

[0121] Retrieve decoded video frames of the specified format from the results cache queue.

[0122] In some embodiments of this disclosure, the device may further include:

[0123] The notification module is used to notify the video processing module that there are currently no decoded video frames that have been decoded and that no decoded video frames of the specified format have been obtained from the result cache queue after each decoding request is received.

[0124] In some embodiments of this disclosure, the device may further include:

[0125] The judgment module is used to determine whether the result flag of the callback parameter indicates successful decoding when the hardware decoder calls back the decoded video frame before retrieving the decoded video frame that has been successfully decoded from the hardware decoder memory. If so, the retrieval module is called to perform the step of retrieving the decoded video frame that has been successfully decoded from the hardware decoder memory.

[0126] In some embodiments of this disclosure, the device may further include:

[0127] The identification module is configured to identify, after each received decoding request, a specified flag included in the decoding request.

[0128] The notification module is configured to notify the hardware decoder to reset after the hardware decoder decodes the to-be-decoded video frame that is currently not decoded.

[0129] In some embodiments of the present disclosure,

[0130] The notification module is further configured to, after notifying the hardware decoder to reset after the hardware decoder decodes the to-be-decoded video frame that is currently not decoded, notify the video processing module that the decoding is completed if the decoding request is not received for a continuous specified number of times.

[0131] In some embodiments of the present disclosure, the apparatus can further include:

[0132] The receiving module 401 is further configured to receive a discard notification sent by the video processing module, the discard notification being used to indicate discarding the decoded video frame that is currently decoded.

[0133] The discarding module is configured to discard the decoded video frame that is currently decoded.

[0134] In some embodiments of the present disclosure, the apparatus can further include:

[0135] The receiving module 401 is further configured to receive a decoding completion notification sent by the video processing module, the decoding completion notification being used to indicate completing the decoding.

[0136] The determining module is configured to determine whether the hardware decoder has been notified to reset after the hardware decoder decodes the to-be-decoded video frame that is currently not decoded.

[0137] The notification module is configured to notify the hardware decoder to reset after the hardware decoder decodes the to-be-decoded video frame that is currently not decoded if the determination result of the determining module is negative.

[0138] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0139] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0140] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0141] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0142] The computing unit 501 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The computing unit 501 performs various methods and processes described above, such as a video decoding method. For example, in some embodiments, the video decoding method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded onto the RAM 503 and executed by the computing unit 501, one or more steps of the video decoding method described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the video decoding method by any other suitable means, such as by means of firmware.

[0143] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0144] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0145] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0147] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0148] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0149] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0150] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for video decoding, comprising: receiving a decoding request periodically sent by a video processing module; upon receiving each decoding request, parsing a to-be-decoded video frame from the decoding request, and calling a hardware decoder to decode the to-be-decoded video frame; asynchronously obtaining a decoded video frame that has been completed decoding, and returning the decoded video frame that has been completed decoding to the video processing module; after the step of upon receiving each decoding request, the method further comprises: identifying a specified flag included in the decoding request; when the specified flag is identified for the first time to indicate that the to-be-decoded video frame has been sent to the end, notifying the hardware decoder to reset after the to-be-decoded video frame that has not been decoded is decoded; wherein the specified flag in the decoding request sent by the video processing module and including the last to-be-decoded video frame and other decoding requests sent after the decoding request all indicate that the to-be-decoded video frame has been sent to the end. 2.The method of claim 1, further comprising: when being called back by the hardware decoder, obtaining the decoded video frame that has been completed decoding from the hardware decoder memory; packaging the decoded video frame that has been completed decoding into a specified format, and storing the decoded video frame in the specified format into a result cache queue, the specified format being a format supported by the video processing module; the step of obtaining a decoded video frame that has been completed decoding comprises: obtaining the decoded video frame in the specified format from the result cache queue. 3.The method of claim 2, after the step of upon receiving each decoding request, the method further comprises: if the decoded video frame in the specified format is not obtained from the result cache queue, notifying the video processing module that there is no decoded video frame that has been completed decoding. 4.The method of claim 2, before the step of obtaining the decoded video frame that has been completed decoding from the hardware decoder memory, the method further comprises: when being called back by the hardware decoder, judging whether a result flag of a callback parameter indicates decoding success; if yes, performing the step of obtaining the decoded video frame that has been completed decoding from the hardware decoder memory. 5.The method of claim 1, after the step of notifying the hardware decoder to reset after the to-be-decoded video frame that has not been decoded is decoded, the method further comprises: if the decoded video frame that has been completed decoding is not obtained after receiving decoding requests for a continuous specified number of times, notifying the video processing module that decoding has been completed, so that the video processing module stops sending decoding requests. 6.The method of any one of claims 1-5, further comprising: receiving a discard notification sent by the video processing module, the discard notification being used to indicate discarding the decoded video frame that has been completed decoding; discarding the decoded video frame that has been completed decoding. 7.The method of any one of claims 1-5, further comprising: receiving a decoding completion notification sent by the video processing module, the decoding completion notification being used to indicate completing decoding. determining whether the hardware decoder has been notified to reset after decoding a current undecoded video frame to be decoded; if not, notifying the hardware decoder to reset after decoding the current undecoded video frame to be decoded.

8. A video decoding apparatus, comprising: a receiving module configured to receive a decoding request periodically sent by a video processing module; a calling module configured to call a hardware decoder to decode a video frame to be decoded parsed from the decoding request each time a decoding request is received; a returning module configured to asynchronously acquire a decoded video frame that has been completed decoding, and return the decoded video frame that has been completed decoding to the video processing module; the apparatus further comprises: an identifying module configured to identify a specified flag included in the decoding request each time a decoding request is received; a notifying module configured to notify the hardware decoder to reset after decoding a current undecoded video frame to be decoded when the specified flag is identified for the first time to indicate that the video frame to be decoded has been sent to the end, wherein the specified flag in the decoding request including the last video frame to be decoded and other decoding requests sent after the decoding request all indicate that the video frame to be decoded has been sent to the end.

9. The apparatus of claim 8, further comprising: an acquiring module configured to acquire the decoded video frame that has been completed decoding from a memory of the hardware decoder when the hardware decoder is called back; a storing module configured to encapsulate the decoded video frame that has been completed decoding into a specified format, and store the decoded video frame in the specified format into a result cache queue, wherein the specified format is a format supported by the video processing module; the returning module is specifically configured to: acquire the decoded video frame in the specified format from the result cache queue.

10. The apparatus of claim 9, further comprising: a notifying module configured to notify the video processing module that there is no decoded video frame that has been completed decoding if the decoded video frame in the specified format is not acquired from the result cache queue each time a decoding request is received.

11. The apparatus of claim 9, further comprising: a judging module configured to judge whether a result flag of a calling parameter indicates decoding success when the hardware decoder is called back before the decoded video frame that has been completed decoding is acquired from the memory of the hardware decoder; if yes, calling the acquiring module to execute the step of acquiring the decoded video frame that has been completed decoding from the memory of the hardware decoder.

12. The apparatus of claim 8, the notifying module is further configured to notify the video processing module that decoding is completed if the decoded video frame that has been completed decoding is not acquired after a continuous specified number of decoding requests are received after the hardware decoder is notified to reset after decoding the current undecoded video frame to be decoded, so that the video processing module stops sending decoding requests.

13. The apparatus of any one of claims 8-12, further comprising: The receiving module is further configured to receive a discard notification sent by the video processing module, the discard notification being used to indicate discarding of the decoded video frame that has been completed decoding; The discarding module is configured to discard the decoded video frame that has been completed decoding.

14. The apparatus according to any one of claims 8-12, further comprising: The receiving module is further configured to receive a decoding end notification sent by the video processing module, the decoding end notification being used to indicate ending of decoding; The determining module is configured to determine whether the hardware decoder has been notified to reset after decoding of the to-be-decoded video frame that has not been decoded is completed; The notifying module is configured to notify the hardware decoder to reset after decoding of the to-be-decoded video frame that has not been decoded is completed, if the determination result of the determining module is negative.

15. An electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method in any one of claims 1-7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method in any one of claims 1-7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method in any one of claims 1-7.

Images