Video streaming live broadcast method, device and storage medium
By directly converting and saving image data to the CPU cache during decoding of the video player, the problems of low video streaming live broadcast efficiency and high CPU usage in the prior art are solved, and more efficient video streaming live broadcast is achieved.
Patent Information
- Application Number
- CN202210947845.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-09
- Publication Date
- 2025-01-14
- Estimated Expiration
- 2042-08-09
AI Technical Summary
In the prior art, the conversion efficiency of live video streaming broadcasting is low, resulting in high occupancy rate and large power consumption on the receiving terminal CPU, and is not suitable for real-time streaming of dynamic picture data.
When the video player decodes the video, the decoded image data is directly converted into the image data in the format required for live broadcast and saved it in the CPU pixel cache to avoid the cumbersome steps of the CPU copying and converting from the GPU cache. IJK The player or ffplay/VLC player converts the YUV format data of AVFrame type to the ARGB format data of CVPixelBufferRef type.
It improves the efficiency of live video streaming, reduces CPU performance, reduces power consumption, and is suitable for real-time streaming of dynamic picture data.
Smart Images
Figure CN115243069B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video live broadcast technology, and in particular to a video live streaming method, device and storage medium. Background Art
[0002] Online live streaming has become one of the main interactive scenarios for people in the Internet age. With the development of live streaming technology, today's live streaming rooms can not only broadcast real-time images, but also stream videos stored in live streaming devices to the live streaming rooms as live content.
[0003] In the prior art, the way to push the video to the live broadcast room is to convert the image screen (view) output by the video player into pixel buffer data (pixelBuffer) for live streaming. This method relies on the CoreGraphics library of iOS, and uses the renderInContext method of the video view.layer to copy the video screen data from the GPU cache to the CPU cache and process it into pixelBuffer. The bus data transmission is time-consuming and the conversion efficiency is low. It is only suitable for occasional image generation by the app, such as screenshots. However, since the video screen is a dynamic screen, data transmission is time-consuming and has a high CPU occupancy rate and high power consumption on the receiving terminal. Summary of the invention
[0004] In order to solve the deficiencies in the prior art, the purpose of this application is to provide a video live streaming method, device and storage medium, which directly processes the video data decoded in the video player to generate the picture data required for streaming, thereby improving the streaming efficiency.
[0005] To achieve the above-mentioned purpose, the present application provides a video streaming live broadcast method, which is used for an IOS terminal to stream a video to a live broadcast, and the method includes:
[0006] When the video player decodes the live video to be pushed into image data in a first format for playback and display, the video player synchronously converts the image data in the first format into image data in a second format required for live broadcast and saves the converted image data in a CPU pixel cache;
[0007] The live streaming end periodically obtains the image data in the second format from the CPU pixel cache and streams it to the live streaming room, while the player plays and displays the image data in the first format;
[0008] The video player is an ijk player, the image data in the first format is image data in the YUV format of the AVFrame type, and the image data in the second format is image data in the ARGB format of the CVPixelBufferRef type.
[0009] Furthermore, the video player may be any one of ffplay player and VLC player.
[0010] Furthermore, it also includes creating a CPU pixel cache, wherein the CPU pixel cache is created based on the width, height and pixel format of the live broadcast picture.
[0011] Furthermore, the step of the live streaming end periodically obtaining the image data in the second format from the CPU pixel cache also includes: setting a timer according to the live streaming FPS requirement to periodically obtain the image data in the second format from the CPU pixel cache and stream it to the live streaming.
[0012] To achieve the above-mentioned purpose, the present application also provides a video streaming live broadcast device, which is used for an IOS terminal to stream a video to a live broadcast, including a video player, wherein the video player includes a decoding unit, and the decoding unit is used to decode the video to be streamed live into image data in a first format for playback by the video player;
[0013] The video player further includes a conversion unit, which is used to convert the image data in the first format into image data in a second format required for live streaming;
[0014] The video player is an ijk player, the image data in the first format is image data in the YUV format of the AVFrame type, and the image data in the second format is image data in the ARGB format of the CVPixelBufferRef type.
[0015] Furthermore, the conversion unit is configured in the decoding unit.
[0016] To achieve the above object, the present application also provides an electronic device, including:
[0017] processor;
[0018] a memory including one or more computer program modules;
[0019] Among them, the one or more computer program modules are stored in the memory and are configured to be executed by the processor, and the one or more computer program modules include modules for implementing the above-mentioned video live streaming method.
[0020] To achieve the above-mentioned purpose, the present application also provides a computer storage medium, including a computer program carried on a non-transitory computer-readable medium, wherein the computer program includes a method for executing the above-mentioned video live streaming method.
[0021] Compared with the prior art, this application has the following beneficial effects:
[0022] This application directly converts the decoded image data into image data in the format required for live broadcast when the video player decodes the video, avoiding the tedious steps of the CPU copying the image data from the GPU cache and then performing the conversion, thereby improving the conversion efficiency and reducing the CPU performance occupation.
[0023] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or may be understood by practicing the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same reference numerals represent the same elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0025] Figure 1 A flow chart of a video live streaming method provided for one embodiment of the present application;
[0026] Figure 2 This is a schematic diagram of image data format conversion according to an embodiment of the present application;
[0027] Figure 3 This is a schematic diagram of image data format conversion in another embodiment of the present application;
[0028] Figure 4 A schematic diagram of the structure of a video live streaming device provided for one embodiment of the present application;
[0029] Figure 5 A schematic block diagram of an electronic device provided for one embodiment of the present application;
[0030] Figure 6 A schematic diagram of a storage medium provided for an embodiment of the present application;
[0031] Figure 7 The figure is a flow chart of a live streaming video method in the prior art. DETAILED DESCRIPTION
[0032] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.
[0033] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.
[0034] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0035] It should be noted that the modifications of "one" and "plurality" mentioned in this application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more". "Plurality" should be understood as two or more.
[0036] Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings.
[0037] Figure 1 A flow chart of a video live streaming method provided in an embodiment of the present application is provided below. Figure 1 , the video live streaming method of the present application is described in detail. In at least one embodiment of the present application, the method at least includes step S101 and step S102:
[0038] First, in step S101, when the video player decodes the live video to be pushed into image data in a first format for playback and display, the video player synchronously converts the image data in the first format into image data in a second format required for live broadcast and saves it to the CPU pixel cache;
[0039] In step S102, the live streaming end periodically obtains the image data in the second format from the CPU pixel cache and streams it to the live streaming room, while the player plays and displays the image data in the first format;
[0040] The video player is an ijk player, the image data in the first format is image data in the YUV format of the AVFrame type, and the image data in the second format is image data in the ARGB format of the CVPixelBufferRef type.
[0041] In order to better understand the technical solution of the present invention, the prior art is now described. Figure 7 The flowchart of the live streaming video method of the prior art is as follows: Figure 7 As shown, the live streaming video method in the prior art includes:
[0042] S701: The video player decodes the live video to be pushed into image data in a first format for playback and display;
[0043] S702: The video player sends the image data in the first format to the GPU for rendering and playback.
[0044] S703: The live streaming end periodically obtains the display screen data of the player from the GPU cache to the CPU cache, converts it into the image data in the second format required for live streaming, and saves it to the CPU pixel cache;
[0045] S704: The live streaming end periodically obtains the image data in the second format from the CPU pixel cache and streams it to the live streaming.
[0046] It can be seen that the technical solution in the prior art, from the video to be pushed to the image data in the second format required for live broadcast, also goes through the process of the video player decoding into the video data in the first format, the GPU rendering to generate display screen data, and obtaining the display screen data from the GPU and converting it in the CPU to the image data in the second format required for live broadcast and saving it to the CPU. Compared with the technical solution of the present application, the two steps of sending the video data in the first format to the GPU for rendering as display screen data and obtaining the display screen data from the GPU cache to convert it into the image data in the second format required for live broadcast and saving it to the CPU are reduced; instead, during decoding, the decoded image data in the first format is converted into the image data in the second format required for live broadcast, thereby improving the streaming speed.
[0047] It should be noted that the ijk player is an open source player. In other embodiments of the present application, any other open source player such as ffplay and VLC player can also be used.
[0048] Preferably, the method further includes creating a CPU pixel cache; the CPU pixel cache is created based on the width, height and pixel format of the live broadcast picture.
[0049] Furthermore, the step of the live streaming end periodically obtaining the image data in the second format from the CPU pixel cache also includes: setting a timer according to the live streaming FPS requirement to periodically obtain the image data in the second format from the CPU pixel cache and stream it to the live streaming.
[0050] Figure 2 This is a schematic diagram of image data format conversion in an embodiment of the present application, such as Figure 2As shown, according to the width (width) and height (height) of the live screen, the pixel format creates a CPU pixel buffer (PixelBuffer) through CVPixelBufferCreate of IOS, and uses the copyAVFrameToPixelBuffer function to convert the AVframe type image data avframe decoded by the ijk player into the CVPixelBufferRef type image data cvImage and save it in the CPU pixel buffer.
[0051] It can be understood that the above-mentioned method for converting the image data format is configured in the decoding method of the ijk player, that is, its corresponding code is placed in the decoding function of the ijk player.
[0052] Figure 3 This is a schematic diagram of image data format conversion in another embodiment of the present application, such as Figure 3 As shown, it converts the image data avframe of the AVFrame type in the YUV storage format parsed by the ijk player into the image data cvImage of the CVPixelBufferRef type in the ARGB storage format and saves it in the CPU pixel cache.
[0053] Figure 4 A schematic diagram of a video streaming live broadcast device provided in an embodiment of the present application, wherein the video streaming live broadcast device is used for an IOS terminal to stream a video to a live broadcast, such as Figure 4 As shown, it includes a video player 41, and the video player includes a decoding unit 42, and the decoding unit 42 is used to decode the video to be pushed live into image data in a first format for playback by the video player 41;
[0054] The video player 41 further includes a conversion unit 43, which is used to convert the image data in the first format into image data in the second format required for live streaming;
[0055] The video player 41 is an ijk player, the image data in the first format is image data in the YUV format of the AVFrame type, and the image data in the second format is image data in the ARGB format of the CVPixelBufferRef type.
[0056] Preferably, the conversion unit 43 is configured in the decoding unit 42 .
[0057] Figure 5 A schematic block diagram of an electronic device provided in one embodiment of the present application. Figure 5As shown, the electronic device 14 includes a processor 141 and a memory 142. The memory 142 is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 141 is used to run non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are run by the processor 141, one or more steps of the video live streaming method described above can be executed. The memory 142 and the processor 141 can be interconnected through a bus system and / or other forms of connection mechanisms (not shown in the figure).
[0058] For example, the processor 141 may be a central processing unit (CPU), a digital signal processor (DSP), or other forms of processing units with data processing capabilities and / or program execution capabilities, such as a field programmable gate array (FPGA), etc.; for example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor 141 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 14 to perform desired functions.
[0059] For example, the memory 142 may include any combination of one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 141 may run one or more computer program modules to implement various functions of the electronic device 14. Various applications and various data, as well as various data used and / or generated by the application, etc. may also be stored in the computer-readable storage medium.
[0060] It should be noted that in the embodiments of the present application, the specific functions and technical effects of the electronic device 14 can be referred to the description of the method above, and will not be repeated here.
[0061] Figure 6 A schematic diagram of a storage medium provided for one embodiment of the present application. Figure 6 As shown, the storage medium 150 is used to store non-transitory computer-readable instructions 151. For example, when the non-transitory computer-readable instructions 151 are executed by a computer, one or more steps in the video live streaming method described above can be executed.
[0062] For example, the storage medium 150 may be applied to the electronic device 14. Figure 5 The memory 142 in the electronic device 14 is shown. For example, the description of the storage medium 150 can be referred to Figure 5 The corresponding description of the memory 142 in the electronic device 14 shown is not repeated here.
[0063] It should be noted that the storage medium (computer-readable medium) mentioned above in the present application may be a computer-readable signal medium or a non-transitory computer-readable storage medium or any combination of the above two. The non-transitory computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of non-transitory computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a non-transitory computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in conjunction with an instruction execution system, device or device.
[0064] In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a non-temporary computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0065] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0066] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0067] In the above, combined with Figures 1 to 6 The video push streaming live broadcast method, electronic device and storage medium provided by the embodiment of the present application are described. The video push streaming live broadcast method provided by the embodiment of the present application converts the decoded image data into image data in the format required for live broadcast, avoiding the tedious steps of the CPU copying the image data from the GPU cache and then performing the conversion, thereby improving the efficiency of the conversion.
[0068] The above description is only a partial embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
[0069] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0070] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A video streaming live broadcast method, used for an IOS terminal to stream a video to a live broadcast, the method comprising: When the video player decodes the live video to be pushed into image data in a first format for playback and display, the video player synchronously converts the image data in the first format into image data in a second format required for live broadcast and saves the converted image data into a CPU pixel cache, where the CPU pixel cache is created based on the width, height and pixel format of the live screen; The live streaming end periodically obtains the image data in the second format from the CPU pixel cache and streams it to the live streaming room, while the player plays and displays the image data in the first format; The video player is an ijk player, the image data in the first format is image data in the YUV format of the AVFrame type, and the image data in the second format is image data in the ARGB format of the CVPixelBufferRef type.
2. The video live streaming method according to claim 1, characterized in that: The video player may be any one of ffplay player and VLC player.
3. The video live streaming method according to claim 1, characterized in that: The step of the live streaming end periodically obtaining the image data in the second format from the CPU pixel cache and streaming it to the live streaming also includes: setting a timer according to the live streaming FPS requirement to periodically obtain the image data in the second format from the CPU pixel cache and streaming it to the live streaming.
4. A video streaming live broadcast device, used for an IOS terminal to stream a video to live broadcast, comprising a video player, the video player comprising a decoding unit, the decoding unit being used to decode the video to be streamed live into image data in a first format for playback by the video player, characterized in that: The video player further includes a conversion unit, which is used to convert the image data in the first format into image data in a second format required for live streaming; The video player is an ijk player, the image data in the first format is image data in the YUV format of the AVFrame type, and the image data in the second format is image data in the ARGB format of the CVPixelBufferRef type.
5. The video live streaming device according to claim 4, characterized in that: The conversion unit is configured in the decoding unit.
6. An electronic device, characterized in that: include: processor; a memory including one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include a method for implementing the video live streaming method according to any one of claims 1 to 3.
7. A computer storage medium, characterized in that: It includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a method for executing the video live streaming method according to any one of claims 1 to 3.