A method, device and storage medium for quickly generating a first image of a video stream

Abstract

The application discloses a kind of video stream first frame image fast generation method, device and storage medium, method includes the following steps: ACodec layer obtains source video stream, and video parameter is obtained from source video data analysis;The video parameter specifically includes video width, video height and the target buffer number required;ACodec layer transmits video parameter to openMax layer, and openMax layer establishes target buffer according to the video parameter;Decoder is distributed to the target buffer by openMax layer;Decoder decodes source video stream in target buffer, and obtains decoded video stream;Video stream first frame image is extracted from decoded video stream by openMax layer, and video stream first frame image is transmitted to display module and is displayed.The application can obtain video stream parameter information in advance by adding parameter pre-configuration means, and only needs to establish buffer once in video stream decoding process, reduces first frame image loading time consumption, achieves the purpose of fast drawing.

Description

Technical Field

[0001] This invention relates to the field of video decoding technology, and in particular to a method, apparatus and storage medium for rapidly generating the first frame image of a video stream. Background Technology

[0002] OpenMax is an open standard released by the Khronos organization for achieving efficient multimedia acceleration. In screen mirroring scenarios, receiving devices such as set-top boxes need to decode and display video stream data after receiving it from the sending end (such as a mobile phone). The mainstream Android multimedia path used for this is MediaCodec. When this path processes the received video stream data, OpenMax is needed to start the corresponding decoder. After the decoder decodes the video stream data, OpenMax sends it to the display module for rendering and display.

[0003] Within the MediaCodec framework, parameter configuration before video stream display needs to be performed at the Acodec layer, where the Acodec layer initializes the display based on the video parameters. However, in existing technologies, the actual video parameters are only obtained after the decoder has decoded the video, but the output buffer is established before the decoder decodes. This means that after the decoder decodes the first frame of the video stream, the first frame cannot be output because the buffer parameters and the actual video parameters do not match. The Acodec layer needs to rebuild the buffer based on the video parameters before the first frame can be output correctly, resulting in excessively long loading times for the first frame. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus and storage medium for rapidly generating the first frame image of a video stream.

[0005] A first aspect of the present invention provides a method for rapidly generating the first frame image of a video stream, comprising the following steps: The ACodec layer obtains the source video stream and parses the video parameters from the source video data; the video parameters specifically include the video width, video height, and the number of target buffers required. The ACodec layer transmits video parameters to the OpenMax layer, and the OpenMax layer establishes a target buffer based on the video parameters. The OpenMax layer assigns the decoder to the target buffer; the decoder decodes the source video stream in the target buffer to obtain the decoded video stream; The first frame of the video stream is extracted from the self-decoding video stream in the OpenMax layer and transmitted to the display module for display.

[0006] Furthermore, the ACodec layer parses video parameters from the source video data, which is specifically achieved through the following steps: In the ACodec layer, set the setPortRefFrameNum() interface function, and call the getParameter() interface function through the setPortRefFrameNum() interface. The ACodec layer obtains the video width and height from the source video stream by parsing the getParameter() interface function. The ACodec layer's setPortRefFrameNum() interface function parses the required number of target buffers from the source video stream.

[0007] Furthermore, the ACodec layer transmits video parameters to the openMax layer, specifically through the following steps: The ACodec layer calls the setParameter() interface function through the setPortRefFrameNum() interface; The ACodec layer transmits the video width, video height, and number of target buffers to the openMax layer through the setParameter() interface function.

[0008] Furthermore, the OpenMax layer transmits the first frame of the video stream to the display module for display, specifically through the following steps: Set the .drain interface function in the ACodec layer; The OpenMax layer transmits the first frame of the video stream to the display module via the .drain interface function.

[0009] Furthermore, the video parameters also include the number of reference frames, and the method further includes the following steps: The ACodec layer obtains the number of reference frames in the source video stream and transmits the number of reference frames in the source video stream to the openMax layer. The OpenMax layer sets the number of reference frames of the source video stream on the decoder, and the decoder decodes the source video stream according to the number of reference frames.

[0010] Furthermore, after obtaining the number of reference frames of the source video stream at the ACodec layer, the following steps are also included: The ACodec layer calls the findInt32() interface function to assign the number of reference frames in the source video stream.

[0011] Furthermore, it also includes the following steps: The OpenMax layer checks the target buffer parameters to determine whether the target buffer parameters are compatible with the hardware information of the decoder and the display module. When the target buffer parameters are not compatible with the decoder's hardware information, the target buffer parameters are aligned according to the decoder's hardware information. When the target buffer parameters are not compatible with the hardware information of the display module, the hardware information of the display module is modified according to the target buffer parameters.

[0012] A second aspect of this invention discloses a device for rapidly generating the first frame image of a video stream, comprising an ACodec layer, an openMax layer, and a decoder; wherein: The ACodec layer is used to obtain the source video stream and parse the video parameters from the source video data; the video parameters specifically include the video width, video height, and the number of target buffers required. The ACodec layer is also used to transmit video parameters to the openMax layer, which then establishes a target buffer based on the video parameters. The openMax layer is used to allocate decoders to the target buffer; the decoders are used to decode the source video stream in the target buffer to obtain the decoded video stream. The openMax layer is also used to extract the first frame image of the video stream from the self-decoding video stream and transmit the first frame image of the video stream to the display module for display.

[0013] A third aspect of the present invention discloses a computer-readable storage medium storing a program that is executed by a processor to implement a method for rapidly generating the first frame image of a video stream.

[0014] The embodiments of this invention offer the following beneficial effects: By incorporating parameter pre-configuration, this invention enables the OpenMax layer to obtain video stream parameter information in advance. During video stream decoding, only one buffer needs to be established, reducing the loading time of the first frame and achieving rapid image output. The buffer applied for and set in this invention can meet the hardware alignment requirements of the decoder and display module, and can be widely applied to video streaming service platforms such as security cameras, set-top boxes, and projectors.

[0015] Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description or may be learned by practice of the invention. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is the video stream decoding process of the existing MediaCodec framework; Figure 2 This is a flowchart of the video stream decoding process of a method, apparatus and storage medium for rapidly generating the first frame image of a video stream according to the present invention; Figure 3 This is an interface function call diagram of a method, apparatus, and storage medium for rapidly generating the first frame image of a video stream according to the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] MediaCodec is a codec component in Android that can encode and decode common audio and video formats such as H.264, H.265, AAC, and 3GP. The MediaCodec playback framework typically uses OpenMax for data transmission and processing, including retrieving the first frame of the video from the decoder and sending it to the display module for display. The existing video stream decoding process of the MediaCodec framework is as follows: Figure 1 As shown, the source video stream is obtained through the network module, and the decoding of the source video stream is completed through six function calls within the MediaCodec framework, outputting the first frame image to the display module. The detailed explanation of the function call process is as follows: ①: Default parameter initialization process: After the Acodec layer reads the configuration file, it calls the initialization process of the openMax layer through process 1. The width and height of the output buffer and the number of reference frames are default parameters, not the actual parameters of the source video stream. ②: After the openMax layer returns the confirmation information to the Acodec layer through process 2, it allocates and sets the first output buffer according to the default parameters; ③: The OpenMax layer sets the first output buffer to the decoder through process 3, and the decoder begins to decode the video parameter information of the source video stream; ④: After the decoder decodes the video parameter information of the source video stream, the OpenMax layer obtains the actual video parameters of the output buffer required by the source video stream through process 4, and thus finds that the video parameter information of the source video stream is inconsistent with the parameter information set in the first output buffer. ⑤: The OpenMax layer calls back the actual video parameters to the Acodec layer through process 5; ⑥: The Acodec layer allocates and sets the second output buffer based on the actual video parameters; ⑦: The OpenMax layer sets the first output buffer to the decoder through process 7, and the decoder outputs the first frame of the video stream normally.

[0020] As can be seen from the above, due to the existence of a two-step buffer setting in the existing technology, the output of the first frame of the video stream is slow, usually more than 300ms, which cannot meet the user experience.

[0021] To accelerate the output speed of the first frame of a video stream, this embodiment provides a method for rapidly generating the first frame image of a video stream, including the following steps: S100: The ACodec layer obtains the source video stream and parses the video parameters from the source video data; the video parameters specifically include the video width, video height, and the number of target buffers required. The S200:ACodec layer transmits video parameters to the openMax layer, and the openMax layer establishes a target buffer based on the video parameters. The S300:openMax layer allocates the decoder to the target buffer; the decoder decodes the source video stream in the target buffer to obtain the decoded video stream; The S400:openMax layer extracts the first frame image of the video stream from the self-decoding video stream and transmits the first frame image of the video stream to the display module for display.

[0022] The interface function call diagram in this embodiment is referenced. Figure 3 The following details the interface functions called and the operations performed in each step of this embodiment.

[0023] S100: The ACodec layer obtains the source video stream and parses the video parameters from the source video data.

[0024] In this embodiment, the video parameters specifically include video width, video height, and the number of target buffers required. The video parameters are mainly used to request and set the target buffers. When the ACodec layer does not obtain the video parameters of the source video stream, it will set the target buffers according to the default parameters, causing the decoder to be unable to output the first frame image. This embodiment obtains the video parameter information before the first round of buffer setting in the ACodec layer, which can avoid the second round of buffer setting in the ACodec layer and save the image output time.

[0025] In this embodiment, video parameters are parsed from the source video data in the ACodec layer, which is achieved through the following steps: S101. Set the setPortRefFrameNum() interface function in the ACodec layer, and call the getParameter() interface function through the setPortRefFrameNum() interface. In this embodiment, a new setPortRefFrameNum() interface function is set in the ACodec layer to enable the ACodec layer to call the getParameter() interface function, thereby parsing the video parameters from the source video data.

[0026] The S102.ACodec layer parses the video width and height from the source video stream using the getParameter() interface function. The getParameter() interface function is a standard interface function of the MediaCodec pathway on the Android platform, used to obtain basic video parameter information such as width, height, and audio information.

[0027] The S103.ACodec layer's setPortRefFrameNum() interface function parses the required number of target buffers from the source video stream. Another function of the setPortRefFrameNum() interface function is to parse the required number of target buffers from the source video stream. The ACodec layer can obtain the video parameter information required to establish the target buffers through the setPortRefFrameNum() and getParameter() interface functions, and pass it to the openMax layer to establish target buffers that match the source video stream.

[0028] The S200:ACodec layer transmits video parameters to the openMax layer, which then establishes a target buffer based on the video parameters.

[0029] In this embodiment, the ACodec layer transmits video parameters to the OpenMax layer, which is achieved through the following steps: The S201.ACodec layer calls the setParameter() interface function through the setPortRefFrameNum() interface. Similar to the getParameter() interface function, the setParameter() interface function is also a standard interface function of the MediaCodec channel on the Android platform. It is used to set basic video parameters such as width, height, and audio information, and can be called through the setPortRefFrameNum() interface function.

[0030] The S202.ACodec layer transmits the video width, video height, and number of target buffers to the openMax layer via the setParameter() interface function. OpenMax then obtains the video data information related to the source video stream and directly establishes a target buffer that matches the source video stream.

[0031] The S300: OpenMax layer allocates the decoder to the target buffer; the decoder decodes the source video stream in the target buffer to obtain the decoded video stream. The decoder (vdecoder, video decoder) is used to decode the source video stream, after which the video image and audio can be played on the display module. When the decoder is set in the buffer, it can decode the source video stream. Furthermore, since the parameter information of the target buffer corresponds to the parameter information of the source video stream, the first frame image obtained from the decoding can be directly output to the display module without needing to rebuild the buffer.

[0032] The S400:openMax layer extracts the first frame image of the video stream from the self-decoding video stream and transmits the first frame image of the video stream to the display module for display.

[0033] In this embodiment, the OpenMax layer transmits the first frame of the video stream to the display module for display, which is achieved through the following steps: S401. Set the .drain interface function in the ACodec layer; the .drain interface function is used to establish a video output channel in the ACodec layer, so that video images can be output to the display module.

[0034] The S402.openMax layer transmits the first frame of the video stream to the display module via the .drain interface function.

[0035] In some implementations, such as screen projection devices like set-top boxes, the sending end encodes the source video stream into AVC format. When decoding the source video stream in AVC format, it is necessary to further obtain the number of reference frames of the video in order to complete the decoding work normally.

[0036] Therefore, in this part of the embodiment, the video parameters also include the number of reference frames, refFrameNum, and the method for quickly generating the first frame image of the video stream further includes the following steps: The S501.ACodec layer obtains the number of reference frames in the source video stream and transmits the number of reference frames in the source video stream to the openMax layer. The S502.openMax layer sets the number of reference frames of the source video stream on the decoder, and the decoder decodes the source video stream according to the number of reference frames.

[0037] In some embodiments, after obtaining the number of reference frames of the source video stream at the ACodec layer, the following steps are also included: The S503.ACodec layer calls the findInt32() interface function to assign a value to the number of reference frames in the source video stream. In this embodiment, it is mainly used to initialize the number of reference frames (refFrameNum) after the number of reference frames in the source video stream has been passed, that is, to set the value of refFrameNum to -1 to achieve the initialization purpose.

[0038] In some embodiments, the method for rapidly generating the first frame of a video stream further includes the following steps: The S600.openMax layer checks the target buffer parameters to determine whether the target buffer parameters are compatible with the hardware information of the decoder and the display module. S601. When the target buffer parameters are not compatible with the decoder's hardware information, the target buffer parameters are aligned according to the decoder's hardware information. S602. When the target buffer parameters are not compatible with the hardware information of the display module, modify the hardware information of the display module according to the target buffer parameters.

[0039] By aligning the target buffer parameters with the hardware information of the decoder and display module in step S600, the generated target buffer can be adapted to the decoder and display module, and the first frame of the video image can be output normally.

[0040] In some embodiments, alternative solutions can be used to achieve the goal of rapid first-frame output of the present invention. For example, the same effect can be achieved by modifying the OpenMax initialization process or adjusting the default OpenMax parameters. However, the process and steps of obtaining video parameters before generating the buffer are essential, such as setting the number, width, and height of matching reference frames. The present invention has been tested and verified in actual products, proving that compared with existing technologies, it can significantly reduce the first-frame output time and achieve the goal of rapid first-frame output.

[0041] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0042] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0043] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0044] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0045] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0046] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for rapidly generating the first frame image of a video stream, characterized in that, Includes the following steps: The ACodec layer obtains the source video stream and parses the video parameters from the source video data; the video parameters specifically include the video width, video height, and the number of target buffers required. The ACodec layer transmits video parameters to the OpenMax layer, and the OpenMax layer establishes a target buffer based on the video parameters. In the ACodec layer, set the setPortRefFrameNum() interface function, and call the getParameter() interface function through the setPortRefFrameNum() interface. The ACodec layer's setPortRefFrameNum() interface function parses the required number of target buffers from the source video stream; The ACodec layer calls the setParameter() interface function through the setPortRefFrameNum() interface; The ACodec layer transmits the target number of buffers to the openMax layer through the setParameter() interface function; The OpenMax layer assigns the decoder to the target buffer; The decoder decodes the source video stream in the target buffer to obtain the decoded video stream; The first frame of the video stream is extracted from the self-decoding video stream of the OpenMax layer and transmitted to the display module for display. The OpenMax layer checks the target buffer parameters to determine whether the target buffer parameters are compatible with the hardware information of the decoder and the display module. When the target buffer parameters are not compatible with the decoder's hardware information, the target buffer parameters are aligned according to the decoder's hardware information. When the target buffer parameters are not compatible with the hardware information of the display module, the hardware information of the display module is modified according to the target buffer parameters.

2. The method for rapidly generating the first frame image of a video stream according to claim 1, characterized in that, The video parameters are parsed from the source video data in the ACodec layer, which is achieved through the following steps: The ACodec layer obtains the video width and height from the source video stream through the getParameter() interface function.

3. The method for rapidly generating the first frame image of a video stream according to claim 2, characterized in that, The ACodec layer transmits video parameters to the OpenMax layer, which is achieved through the following steps: The ACodec layer transmits the video width and height to the openMax layer through the setParameter() interface function.

4. The method for rapidly generating the first frame image of a video stream according to claim 1, characterized in that, The OpenMax layer transmits the first frame of the video stream to the display module for display, which is achieved through the following steps: Set the .drain interface function in the ACodec layer; The OpenMax layer transmits the first frame of the video stream to the display module via the .drain interface function.

5. The method for rapidly generating the first frame image of a video stream according to claim 1, characterized in that, The video parameters also include the number of reference frames, and the method further includes the following steps: The ACodec layer obtains the number of reference frames in the source video stream and transmits the number of reference frames in the source video stream to the openMax layer. The OpenMax layer sets the number of reference frames of the source video stream on the decoder, and the decoder decodes the source video stream according to the number of reference frames.

6. The method for rapidly generating the first frame image of a video stream according to claim 5, characterized in that, After obtaining the number of reference frames of the source video stream at the ACodec layer, the following steps are also included: The ACodec layer calls the findInt32() interface function to assign the number of reference frames in the source video stream.

7. A device for rapidly generating the first frame image of a video stream, characterized in that, This includes the ACodec layer, the openMax layer, and the decoder; among which: The ACodec layer is used to obtain the source video stream and parse the video parameters from the source video data; the video parameters specifically include the video width, video height, and the number of target buffers required. The ACodec layer is also used to transmit video parameters to the OpenMax layer, which establishes a target buffer based on the video parameters. The ACodec layer sets the `setPortRefFrameNum()` interface function, which calls the `getParameter()` interface function. The `setPortRefFrameNum()` interface function in the ACodec layer parses the required number of target buffers from the source video stream. The ACodec layer then calls the `setParameter()` interface function through `setPortRefFrameNum()`. Finally, the ACodec layer transmits the target buffer count to the OpenMax layer through `setParameter()`. The openMax layer is used to allocate the decoder to the target buffer; the decoder is used to decode the source video stream in the target buffer to obtain the decoded video stream. The OpenMax layer is also used to extract the first frame image of the video stream from the self-decoded video stream and transmit the first frame image of the video stream to the display module for display. The openMax layer is also used to check the target buffer parameters to determine whether the target buffer parameters are compatible with the hardware information of the decoder and the display module. When the target buffer parameters are not compatible with the hardware information of the decoder, the target buffer parameters are aligned according to the hardware information of the decoder. When the target buffer parameters are not compatible with the hardware information of the display module, the hardware information of the display module is modified according to the target buffer parameters.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1-6.

Images