Live-streaming processing method, and device and storage medium

Abstract

Provided in the embodiments of the present disclosure are a live-streaming processing method, and a device and a storage medium. The live-streaming processing method comprises: a first live-streaming video chat client receiving a live stream of a second live-streaming video chat client, and caching the latest video frame of the live stream of the second live-streaming video chat client; when the current video frame of the first live-streaming video chat client is acquired, reading the cached latest video frame; on the basis of a single rendering view, synthesizing the current video frame and the latest video frame into a target video frame according to preset layout information; and displaying the target video frame. In a multi-person video chat live-streaming scenario, the current video frame of a first live-streaming video chat client and the latest video frame of a live stream of another, a second live-streaming video chat client are synthesized into a target video frame, and rendering can be realized only by using a single rendering view, thereby reducing the occupancy of resources, reducing the performance pressure on a live-streaming video chat client, and improving the live-streaming fluency.

Description

Live broadcast processing method, device and storage medium

[0001] This application claims priority to the Chinese invention patent application entitled “Live broadcast processing method, device and storage medium” filed on June 28, 2023, with application number: 202310782876.5. The entire contents of this application are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular to a live broadcast processing method, device, and storage medium. Background Art

[0003] During online live broadcasts, multi-person connection is a method often used by anchors to interact with other anchors. Based on this method, the anchor in the current live broadcast room can invite other anchors to broadcast together.

[0004] However, as the number of hosts in multi-person live broadcast scenarios increases, the live broadcast smoothness and the performance of the live broadcast client are seriously affected, and may even cause severe heat.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide a live broadcast processing method, device, and storage medium to reduce the performance pressure on the live broadcast client in a multi-person live broadcast scenario and improve the live broadcast fluency.

[0007] In a first aspect, an embodiment of the present disclosure provides a live broadcast processing method, which is applied to a first live broadcast client, and the method includes: receiving a live broadcast stream of a second live broadcast client, and caching the latest video frame of the live broadcast stream of the second live broadcast client; when obtaining the current video frame of the first live broadcast client, reading the cached latest video frame; based on a single rendering view, synthesizing the current video frame and the latest video frame into a target video frame according to preset layout information; and displaying the target video frame.

[0008] In a second aspect, an embodiment of the present disclosure provides a live broadcast processing device, which is applied to a first live broadcast client, and the device includes: a receiving unit, which is used to receive the live stream of a second live broadcast client; a cache unit, which is used to cache the latest video frame of the live stream of the second live broadcast client; when obtaining the current video frame of the first live broadcast client, reading the cached latest video frame; a rendering unit, which is used to synthesize the current video frame and the latest video frame into a target video frame based on a single rendering view and preset layout information; and a display unit, which is used to display the target video frame.

[0009] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live broadcast processing method described in the first aspect and various possible designs of the first aspect.

[0010] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the live broadcast processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0011] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the live broadcast processing method as described in the first aspect and various possible designs of the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0013] FIG1 is a diagram illustrating an example scenario of a live broadcast processing method provided by an embodiment of the present disclosure;

[0014] FIG2 is a flow chart of a live broadcast processing method provided by an embodiment of the present disclosure;

[0015] FIG3 is a flow chart of a live broadcast processing method provided by another embodiment of the present disclosure;

[0016] FIG4a is a schematic diagram of an interface of a layout method provided by an embodiment of the present disclosure;

[0017] FIG4 b is a schematic diagram of an interface of a layout method provided by another embodiment of the present disclosure;

[0018] FIG5 is a schematic diagram showing alignment of a layer corresponding to the latest video frame of each live stream with a preset list view control provided by an embodiment of the present disclosure;

[0019] FIG6 is a flowchart of a live broadcast processing method provided by another embodiment of the present disclosure;

[0020] FIG7 is a schematic diagram of a target template provided by an embodiment of the present disclosure;

[0021] FIG8 is a flow chart of a live broadcast processing method provided by another embodiment of the present disclosure;

[0022] FIG9 is a schematic diagram of determining a target template according to an embodiment of the present disclosure;

[0023] FIG10 is a structural block diagram of a live broadcast processing device provided by an embodiment of the present disclosure;

[0024] FIG11 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0026] During online live broadcasts, multi-person connection is a method often used by anchors to interact with other anchors. Based on this method, the anchor in the current live broadcast room can invite other anchors to broadcast together.

[0027] In a multi-player live broadcast scenario, any primary live broadcast client will use different rendering views, GL rendering threads, frame buffer objects, and surface views for the current video frame of the primary live broadcast client and the video frames of the live stream of each secondary live broadcast client. These rendering views, GL rendering thread frame buffer objects, and surface views cannot be reused, thus occupying a large amount of live broadcast client resources. However, as the number of hosts in a multi-player live broadcast scenario increases, the live broadcast smoothness is severely affected, the performance of the live broadcast client is severely affected, and it may even cause severe overheating.

[0028] In order to solve the above technical problems, the present disclosure provides a live broadcast processing method, which receives the live stream of the second live broadcast client through the live broadcast client in a multi-person live broadcast room, and caches the latest video frame of each live stream; when obtaining the current video frame of the first live broadcast client, reads the cached latest video frame; based on a single rendering view, synthesizes the current video frame and the latest video frame into a target video frame according to preset layout information; and displays the target video frame. In a multi-person live broadcast scenario, by synthesizing the current video frame of the first live broadcast client and the latest video frames of the live streams of other second live broadcast clients into a target video frame, rendering can be achieved using only a single rendering view, which reduces resource usage, reduces the performance pressure on the live broadcast client, and improves the smoothness of the live broadcast.

[0029] The application scenario of the live broadcast processing method provided by the present disclosure is shown in Figure 1. The execution subject is the first live broadcast client. The first live broadcast client can receive the live streams of one or more second live broadcast clients and cache the latest video frame of each live stream to the corresponding layer. The first live broadcast client can collect the current video frame as the initial layer OriginLayer. When the first live broadcast client collects the current video frame, it reads the latest video frame of each cached live stream, and synthesizes the current video frame and the latest video frame of each live stream into a target video frame for display.

[0030] It should be noted that the user information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] The live broadcast processing method disclosed herein will be described in detail below with reference to specific embodiments.

[0032] Refer to Figure 2, which is a flowchart of a live broadcast processing method provided by an embodiment of the present disclosure. The method of this embodiment can be applied to any live broadcast client (referred to as the first live broadcast client) in a multi-person live broadcast room. The live broadcast processing method includes:

[0033] S201: Receive a live stream from a second live broadcast and microphone connection client, and cache the latest video frame of the live stream from the second live broadcast and microphone connection client.

[0034] In this embodiment, in a multi-person live broadcast scenario, a first live broadcast client can invite one or more second live broadcast clients to join the broadcast, so that the first live broadcast client's screen and the screens of one or more second live broadcast clients can be viewed simultaneously in the interfaces of the first live broadcast client and the second live broadcast client. In a multi-person live broadcast scenario, the first live broadcast client can receive the live streams of one or more second live broadcast clients. Specifically, one or more second live broadcast clients can upload their respective live streams to the server, and the server can send the live streams of one or more second live broadcast clients to the first live broadcast client.

[0035] Furthermore, the live stream of each second live broadcast client is parsed to obtain the video frames in the live stream, and the latest video frame is cached. Specifically, the latest video frame of each live stream can be drawn and cached in a corresponding layer.

[0036] S202: When obtaining the current video frame of the first live broadcast and microphone connection client, read the cached latest video frame.

[0037] In this embodiment, the first live broadcast client can capture the screen on the first live broadcast client side in real time to obtain the current video frame. Based on the current video frame, the current video frame is driven to synthesize with the latest video frame of the live stream of each second live broadcast client. Since the current video frame captured by the first live broadcast client does not need to be transmitted through the network, the output is relatively stable. Therefore, the current video frame captured by the first live broadcast client is driven by the current video frame captured by the first live broadcast client to ensure the stability of the live screen. When the current video frame captured by the first live broadcast client is obtained, the latest video frame of each cached live stream can be read. Specifically, the latest video frame of each live stream can be read from each layer for synthesis of the target video frame.

[0038] S203 : Based on a single rendering view, synthesize the current video frame and the latest video frame of each live stream into a target video frame according to preset layout information.

[0039] In this embodiment, after obtaining the current video frame captured by the first live broadcast client and the latest video frame of the live stream of each second live broadcast client, the current video frame captured by the first live broadcast client and the latest video frame of each live stream can be synthesized and drawn onto the same texture to obtain a target video frame. A single rendering view (RenderView) is shared to achieve rendering of the current video frame and the latest video frame of each live stream in a single view, without the need to use different rendering views for the current video frame of the first live broadcast client and the video frame of the live stream of each second live broadcast client, thereby reducing the performance pressure on the first live broadcast client. During the synthesis process, the layout of the current video frame and the latest video frame of each live stream can be determined based on preset layout information.

[0040] In addition, when the current video frame and the latest video frame of each live stream are synthesized into a target video frame based on a single rendering view according to preset layout information, only one GL rendering thread is needed, instead of using a GL rendering thread for the current video frame of the first live broadcast client and the video frame of the live stream of each second live broadcast client, which further reduces the performance pressure on the first live broadcast client.

[0041] Optionally, in this embodiment, the target video frame can be rendered into a single frame buffer object (Frame Buffer Object, FBO), avoiding the use of different FBOs to cache the current video frame of the first live broadcast client and the video frame of the live stream of each second live broadcast client, further reducing the performance pressure on the first live broadcast client.

[0042] S204: Display the target video frame.

[0043] In this embodiment, after the target video frame is synthesized and rendered, it is displayed in the display interface of the first live broadcast and microphone connection client.

[0044] Since the current video frame of the first live broadcast client and the latest video frame of the live stream of each second live broadcast client are synthesized into a target video frame, in this embodiment, only one surface view SurfaceView is needed to display the target video frame, without the need to use different SurfaceViews to display the current video frame of the first live broadcast client and the video frame of the live stream of each second live broadcast client, which further reduces the performance pressure on the first live broadcast client.

[0045] Optionally, if the target video frame is rendered into a single FBO, the rendered target video frame cached in the FBO is displayed using a single surface view.

[0046] The live broadcast processing method provided in this embodiment receives the live broadcast stream of the second live broadcast client through the first live broadcast client, and caches the latest video frame of the live broadcast stream of the second live broadcast client; when obtaining the current video frame of the first live broadcast client, reads the cached latest video frame; based on a single rendering view, synthesizes the current video frame and the latest video frame into a target video frame according to preset layout information; and displays the target video frame. In a multi-person live broadcast scenario, by synthesizing the current video frame of the first live broadcast client and the latest video frames of the live streams of other second live broadcast clients into a target video frame, only a single rendering view is needed to achieve rendering, which reduces resource usage, reduces the performance pressure on the live broadcast client, and improves the smoothness of the live broadcast.

[0047] Based on the above embodiment, as shown in FIG3 , the step of synthesizing the current video frame and the latest video frame into a target video frame according to the preset layout information in S203 includes:

[0048] S301: Determine the size and position of a first layer corresponding to the current video frame and a second layer corresponding to the latest video frame in the target video frame according to preset layout information.

[0049] S302: Draw the current video frame into the first layer, draw the latest video frame into the second layer respectively, and synthesize the first layer and the second layer into the target video frame.

[0050] In this embodiment, during the synthesis process, it can be determined how the current video frame of the first live broadcast client and the latest video frame of the live stream of each second live broadcast client are laid out in the target video frame according to the preset layout information. For example, as shown in FIG4a, the first layer (Layer) of the current video frame of the first live broadcast client is located on the left side of the target video frame, displaying the picture of the current video frame of the first live broadcast client, and the second layer (Layer) of the latest video frame of the live stream of each second live broadcast client is arranged in a column vertically on the right side of the target video frame, displaying the picture of each second live broadcast client; for another example, as shown in FIG4b, the first layer (Layer) of the current video frame of the first live broadcast client is displayed in full screen in the target video frame, displaying the picture of the current video frame of the first live broadcast client, and the latest video frame of the live stream of each second live broadcast client is displayed in full screen. The second layer (Layer) is located in the form of a floating window above the current video frame of the first live broadcast and microphone connection client, and the screen of each second live broadcast and microphone connection client floats above the screen of the current video frame; of course, other layout methods can also be used. In each layout method, the size and position of the first layer corresponding to the current video frame and the second layer corresponding to the latest video frame of each live stream can be determined according to the preset layout information in the target video frame, and then the current video frame is drawn to the first layer corresponding to the current video frame, and the latest video frame of each live stream is drawn to the second layer corresponding to the latest video frame of each live stream, so that the layers are synthesized into the target video frame.

[0051] Optionally, before synthesizing the current video frame and the latest video frame into a target video frame based on a single rendering view according to preset layout information in S203, it also includes: obtaining the size and position of each view control in the preset list view control, and using the size and position of any view control as the size and position of the second layer corresponding to the latest video frame of the live stream of any second live broadcast client in the target video frame, and obtaining the preset layout information so that the second layer corresponding to the latest video frame of the live stream is aligned with the view control in the preset list view control.

[0052] In this embodiment, since the target video frame ultimately adopts a single surface view, and each live stream in the target video frame also needs to perform UI interaction, a preset list view control (RecyclerView) can be set in the display interface of the multi-person live broadcast scene. The preset list view control includes multiple view controls (ViewHolder). For example, the preset list view control RecyclerView shown in Figure 5 is a nine-grid list view control, which includes nine view controls ViewHolders. The layers corresponding to the latest video frames of each second live broadcast client live stream are aligned with each view control ViewHolder, so that interaction with the corresponding live stream can be achieved through each view control ViewHolder. In order to achieve the alignment of the second layer corresponding to the latest video frame of the live stream of each second live broadcast client with each view control, the size and position of each view control in the preset list view control can be obtained, and the size and position of each view control can be used as the size and position of the layer corresponding to the latest video frame of each live stream in the target video frame, thereby serving as the preset layout information. During synthesis, the current video frame and the latest video frame of the live stream of each second live broadcast client are synthesized into a target video frame according to the preset layout information, so that the second layer corresponding to the latest video frame of each live stream in the target video frame is aligned with each view control in the preset list view control.

[0053] Based on any of the above embodiments, during synthesis, the current video frame of the first live broadcast client and the latest video frame of the live stream of the second live broadcast client may have a certain overlapping area, so it is necessary to set the display hierarchy value, with the highest hierarchy placed at the top and the lowest hierarchy placed at the bottom. If there is an overlapping area, the video frame at the higher hierarchy covers the video frame at the lower hierarchy. Optionally, as shown in Figure 6, synthesizing the current video frame and the latest video frame into the target video frame according to the preset layout information in S203 includes:

[0054] S401, determining a target template for template testing in a fragment shader rendering pipeline, wherein a depth value at an area corresponding to the current video frame in the target template is a level value corresponding to the current video frame, and a depth value at an area corresponding to the latest video frame is a level value corresponding to the latest video frame;

[0055] S402 : Perform a template test according to the level value of the current video frame, the level value of the latest video frame, and the target template, and synthesize the current video frame and the latest video frame based on the template test result.

[0056] In this embodiment, the rendering pipeline of the OpenGL fragment shader includes a stencil test link. The stencil test determines whether a fragment is retained or discarded based on a template. If a fragment fails the stencil test, the fragment is discarded, that is, the fragment is not drawn. If a fragment passes the stencil test, the fragment is retained. In this embodiment, a target template for stencil testing in the fragment shader rendering pipeline can be pre-configured. The depth value of the current video frame in the target template and the corresponding area of ​​the latest video frame of each second live broadcast client live stream is the hierarchical value zOrder corresponding to each area. As shown in Figure 7, the higher the hierarchical value zOrder, the larger the number, that is, the higher the depth value. For example, the latest video frame of the live stream of the second live broadcast client is suspended above the current video frame of the first live broadcast client, that is, the layer value zOrder of the latest video frame of the live stream of the second live broadcast client is higher than the layer value zOrder of the current video frame of the first live broadcast client. The layer value zOrder of the latest video frame of the live stream of the second live broadcast client can be set to 3, and the layer value zOrder of the current video frame of the first live broadcast client is set to 2. In the target template, the depth value of each position in the corresponding area (inside the box) of the latest video frame of the live stream is 3, and the depth value of each position in the area not covered by the current video frame of the first live broadcast client is 2.

[0057] When synthesizing the current video frame of the first live broadcast client and the latest video frame of the live broadcast stream of the second live broadcast client, a template test is performed based on the level value of the current video frame, the level value of the latest video frame, and the target template. The level value of the current video frame and the position that passes the template test in the latest video frame are rendered and drawn, and the position that fails the template test is not rendered and drawn, and finally the synthesized target video frame is obtained.

[0058] In some embodiments, the specific process of template testing may include:

[0059] Determine the level value at any first position of the current video frame as the actual depth value at the first position, and compare the actual depth value at the first position of the current video frame with the depth value at the corresponding position in the target template; if the actual depth value at the first position is higher than or equal to the depth value at the corresponding position in the target template, the template test of the first position passes, and the first position of the current video frame can be rendered; and if the actual depth value at the first position is lower than the depth value at the corresponding position in the target template, the template test of the first position fails, and the first position of the current video frame is not rendered;

[0060] The level value at any second position of the latest video frame of the live stream of any second live broadcast client is determined as the actual depth value at the second position, and the actual depth value at the second position of the latest video frame is compared with the depth value at the corresponding position in the target template. If the actual depth value at the second position is higher than or equal to the depth value at the corresponding position in the target template, the template test of the second position passes, and the second position of the latest video frame can be rendered; if the actual depth value at the second position is lower than the depth value at the corresponding position in the target template, the template test of the second position fails, and the second position of the latest video frame is not rendered.

[0061] By rendering all first positions that pass the template test in the current video frame and all second positions that pass the template test in the latest video frame of each live stream, a hierarchical relationship is achieved during synthesis. After the positions that pass the template test in the current video frame and the latest video frame of each second live broadcast client live stream are rendered, the synthesized target video frame is obtained.

[0062] Based on the above embodiment, if the level value of the latest video frame of any live stream is higher than the level value of the current video frame, then determining the target template for template testing in the fragment shader rendering pipeline in S401, as shown in FIG8 , may include:

[0063] S501, setting the depth values ​​at all positions of the target template as the level values ​​of the current video frame;

[0064] S502: Update the depth value of the area corresponding to the latest video frame in the target template to the level value of the latest video frame according to the level value of the latest video frame and the preset layout information.

[0065] In this embodiment, the preset level value zOrder of the current video frame is obtained. Assuming that the level value zOrder of the current video frame is 2, the depth values ​​at all positions of the target template are set to the level value of the current video frame, that is, the depth values ​​at all positions of the target template are 2, as shown in the upper side of Figure 9; the preset level value zOrder of the latest video frame of the live stream of any second live broadcast client is obtained. Assuming that the level value zOrder of the latest video frame of the live stream of any second live broadcast client is 3, and the position and size of the latest video frame of the live stream in the target video frame can be obtained from the preset layout information. Therefore, the corresponding area of ​​the latest video frame of the live stream of any second live broadcast client in the target template can be determined, that is, the box area in the figure. Further, the depth value at the corresponding area of ​​the latest video frame of the live stream of any second live broadcast client in the target template is updated to the level value of the latest video frame of the live stream of the second live broadcast client, that is, the depth value of each position in the box area of ​​the target template is updated to 3, as shown in the lower side of Figure 9, and finally the target template is obtained.

[0066] In some embodiments, the target template is configured with a buffer (cache), and S501-S502 are implemented in the buffer of the target template, that is, the depth values ​​at all positions of the target template in the buffer of the target template are set to the level values ​​of the current video frame; in the buffer of the target template, the depth value at the corresponding area of ​​the latest video frame of the live stream of any second live broadcast client in the target template is updated to the level value of the latest video frame of the live stream of any second live broadcast client in the target template.

[0067] The live broadcast processing method, device, and storage medium provided by the embodiments of the present disclosure receive the live stream of the second live broadcast client through the first live broadcast client, and cache the latest video frame of the live stream of the second live broadcast client; when obtaining the current video frame of the first live broadcast client, read the cached latest video frame; based on a single rendering view, synthesize the current video frame and the latest video frame into a target video frame according to preset layout information; and display the target video frame. In a multi-person live broadcast scenario, by synthesizing the current video frame of the first live broadcast client and the latest video frames of the live streams of other second live broadcast clients into a target video frame, only a single rendering view is needed to achieve rendering, which reduces resource usage, reduces the performance pressure on the live broadcast client, and improves the smoothness of the live broadcast.

[0068] Corresponding to the live broadcast processing method described in the above embodiment, FIG10 is a block diagram of the live broadcast processing apparatus provided in an embodiment of the present disclosure. For ease of illustration, only the portions relevant to the embodiment of the present disclosure are shown. Referring to FIG10 , the live broadcast processing apparatus 600 includes: a receiving unit 601, a cache unit 602, a rendering unit 603, and a display unit 604.

[0069] In some embodiments, a receiving unit 601 is used to receive a live stream from a second live broadcast client; a caching unit 602 is used to cache the latest video frame of the live stream from the second live broadcast client; when obtaining the current video frame of the first live broadcast client, the cached latest video frame is read; a rendering unit 603 is used to synthesize the current video frame and the latest video frame into a target video frame based on a single rendering view and preset layout information; and a display unit 604 is used to display the target video frame.

[0070] In one or more embodiments of the present disclosure, when synthesizing the current video frame and the latest video frame into a target video frame according to preset layout information, the rendering unit 603 is used to: determine the size and position of the first layer corresponding to the current video frame and the second layer corresponding to the latest video frame in the target video frame according to the preset layout information; draw the current video frame into the first layer, draw the latest video frame into the second layer respectively, and synthesize the first layer and the second layer into the target video frame.

[0071] In one or more embodiments of the present disclosure, when the rendering unit 603 synthesizes the current video frame and the latest video frame into a target video frame according to preset layout information, it is used to: determine a target template for template testing in the fragment shader rendering pipeline, wherein the depth value at the corresponding area of ​​the current video frame in the target template is the level value corresponding to the current video frame, and the depth value at the corresponding area of ​​the latest video frame is the level value corresponding to the latest video frame; perform template testing according to the level value of the current video frame, the level value of the latest video frame and the target template, and synthesize the current video frame and the latest video frame based on the template test result.

[0072] In one or more embodiments of the present disclosure, when performing a template test based on the level value of the current video frame, the level value of the latest video frame of each live stream, and the target template, the rendering unit 603 is used to: determine the level value at any first position of the current video frame as the actual depth value at the first position, compare the actual depth value at the first position of the current video frame with the depth value at the corresponding position in the target template, and if the actual depth value at the first position is higher than or equal to the depth value at the corresponding position in the target template, the template test of the first position passes; determine the level value at any second position of the latest video frame as the actual depth value at the second position, compare the actual depth value at the second position of the latest video frame with the depth value at the corresponding position in the target template, and if the actual depth value at the second position is higher than or equal to the depth value at the corresponding position in the target template, the template test of the second position passes.

[0073] In one or more embodiments of the present disclosure, when synthesizing the current video frame and the latest video frame based on the template test result, the rendering unit 603 is used to: if the template test of any first position of the current video frame passes, render the first position of the current video frame; if the template test of any second position of the latest video frame passes, render the second position of the latest video frame; after completing the rendering of the positions that pass the template test in the current video frame and the latest video frame, obtain the synthesized target video frame.

[0074] In one or more embodiments of the present disclosure, the level value of the latest video frame is higher than the level value of the current video frame.

[0075] In one or more embodiments of the present disclosure, when determining the target template for template testing in the fragment shader rendering pipeline, the rendering unit 603 is used to: set the depth value at all positions of the target template to the level value of the current video frame; and update the depth value at the corresponding area of ​​the latest video frame in the target template to the level value of the latest video frame according to the level value of the latest video frame and the preset layout information.

[0076] In one or more embodiments of the present disclosure, before the rendering unit 603 synthesizes the current video frame and the latest video frame of each live stream into a target video frame based on a single rendering view and according to preset layout information, it is further used to: obtain the size and position of each view control in the preset list view control, use the size and position of any view control as the size and position of the second layer corresponding to the latest video frame in the target video frame, and obtain the preset layout information so that the second layer corresponding to the latest video frame is aligned with any view control in the preset list view control.

[0077] In one or more embodiments of the present disclosure, when the rendering unit 603 synthesizes the current video frame and the latest video frame into a target video frame based on a single rendering view and preset layout information, it is used to: synthesize the current video frame and the latest video frame into a target video frame based on a single rendering view and a single rendering thread according to preset layout information.

[0078] In one or more embodiments of the present disclosure, when displaying the target video frame, the display unit 604 is configured to: display the target video frame using a single surface view.

[0079] In one or more embodiments of the present disclosure, when the rendering unit 603 synthesizes the current video frame and the latest video frame into a target video frame based on a single rendering view and according to preset layout information, it is also used to: render the target video frame into a single frame buffer object; when the display unit 604 displays the target video frame using a single surface view, it is used to: display the rendered target video frame cached in the frame buffer object using a single surface view.

[0080] The live broadcast processing device provided in this embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar, and will not be repeated here in this embodiment.

[0081] Referring to FIG11 , a schematic diagram of the structure of an electronic device 700 suitable for implementing an embodiment of the present disclosure is shown. The electronic device 700 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG11 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0082] As shown in FIG11 , the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0083] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although FIG11 shows an electronic device 700 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0084] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0085] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0086] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0087] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0088] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0090] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0091] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0092] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0093] In a first aspect, according to one or more embodiments of the present disclosure, a live broadcast processing method is provided, which is applied to a first live broadcast client, and the method includes: receiving a live broadcast stream of a second live broadcast client, and caching the latest video frame of the live broadcast stream of the second live broadcast client; when obtaining the current video frame of the first live broadcast client, reading the cached latest video frame; based on a single rendering view, synthesizing the current video frame and the latest video frame into a target video frame according to preset layout information; and displaying the target video frame.

[0094] According to one or more embodiments of the present disclosure, synthesizing the current video frame and the latest video frame into a target video frame according to preset layout information includes: determining the size and position of a first layer corresponding to the current video frame and a second layer corresponding to the latest video frame in the target video frame according to the preset layout information; drawing the current video frame into the first layer, drawing the latest video frame into the second layer, respectively, and synthesizing the first layer and the second layer into the target video frame.

[0095] According to one or more embodiments of the present disclosure, synthesizing the current video frame and the latest video frame into a target video frame according to preset layout information includes: determining a target template for template testing in a fragment shader rendering pipeline, wherein the depth value at the corresponding area of ​​the current video frame in the target template is the level value corresponding to the current video frame, and the depth value at the corresponding area of ​​the latest video frame is the level value corresponding to the latest video frame; performing a template test according to the level value of the current video frame, the level value of the latest video frame and the target template, and synthesizing the current video frame and the latest video frame based on the template test result.

[0096] According to one or more embodiments of the present disclosure, the template test is performed based on the level value of the current video frame, the level value of the latest video frame and the target template, including: determining the level value at any first position of the current video frame as the actual depth value at the first position, comparing the actual depth value at the first position of the current video frame with the depth value at the corresponding position in the target template, if the actual depth value at the first position is higher than or equal to the depth value at the corresponding position in the target template, the template test of the first position passes; determining the level value at any second position of the latest video frame as the actual depth value at the second position, comparing the actual depth value at the second position of the latest video frame with the depth value at the corresponding position in the target template, if the actual depth value at the second position is higher than or equal to the depth value at the corresponding position in the target template, the template test of the second position passes.

[0097] According to one or more embodiments of the present disclosure, the current video frame and the latest video frame are synthesized based on the template test result, including: if the template test of any first position of the current video frame passes, the first position of the current video frame is rendered; if the template test of any second position of the latest video frame passes, the second position of the latest video frame is rendered; after completing the rendering of the positions that pass the template test in the current video frame and the latest video frame, the synthesized target video frame is obtained.

[0098] According to one or more embodiments of the present disclosure, the level value of the latest video frame is higher than the level value of the current video frame.

[0099] According to one or more embodiments of the present disclosure, determining the target template for template testing in the fragment shader rendering pipeline includes: setting the depth values ​​at all positions of the target template to the level value of the current video frame; and updating the depth value at the area corresponding to the latest video frame in the target template to the level value of the latest video frame according to the level value of the latest video frame and the preset layout information.

[0100] According to one or more embodiments of the present disclosure, before synthesizing the current video frame and the latest video frame into a target video frame based on a single rendering view according to preset layout information, it also includes: obtaining the size and position of each view control in the preset list view control, using the size and position of any view control as the size and position of the second layer corresponding to the latest video frame in the target video frame, and obtaining the preset layout information to align the second layer corresponding to the latest video frame with any view control in the preset list view control.

[0101] According to one or more embodiments of the present disclosure, the synthesizing the current video frame and the latest video frame into a target video frame based on a single rendering view and preset layout information includes: based on a single rendering view and a single rendering thread, synthesizing the current video frame and the latest video frame into a target video frame according to preset layout information.

[0102] According to one or more embodiments of the present disclosure, displaying the target video frame includes: displaying the target video frame using a single surface view.

[0103] According to one or more embodiments of the present disclosure, the synthesizing the current video frame and the latest video frame into a target video frame based on a single rendering view according to preset layout information also includes: rendering the target video frame into a single frame buffer object; the displaying the target video frame using a single surface view includes: displaying the rendered target video frame cached in the frame buffer object using a single surface view.

[0104] In a second aspect, according to one or more embodiments of the present disclosure, a live broadcast processing device is provided, which is applied to a first live broadcast client, and the device includes: a receiving unit, which is used to receive the live stream of a second live broadcast client; a caching unit, which is used to cache the latest video frame of the live stream of the second live broadcast client; when obtaining the current video frame of the first live broadcast client, reading the cached latest video frame; a rendering unit, which is used to synthesize the current video frame and the latest video frame into a target video frame based on a single rendering view and preset layout information; and a display unit, which is used to display the target video frame.

[0105] According to one or more embodiments of the present disclosure, when the rendering unit synthesizes the current video frame and the latest video frame into a target video frame according to preset layout information, it is used to: determine the size and position of the first layer corresponding to the current video frame and the second layer corresponding to the latest video frame in the target video frame according to the preset layout information; draw the current video frame into the first layer, draw the latest video frame into the second layer respectively, and synthesize the first layer and the second layer into the target video frame.

[0106] According to one or more embodiments of the present disclosure, when the rendering unit synthesizes the current video frame and the latest video frame into a target video frame according to preset layout information, it is used to: determine a target template for template testing in a fragment shader rendering pipeline, wherein the depth value at the corresponding area of ​​the current video frame in the target template is the level value corresponding to the current video frame, and the depth value at the corresponding area of ​​the latest video frame is the level value corresponding to the latest video frame; perform a template test according to the level value of the current video frame, the level value of the latest video frame and the target template, and synthesize the current video frame and the latest video frame based on the template test result.

[0107] According to one or more embodiments of the present disclosure, when performing a template test based on the level value of the current video frame, the level value of the latest video frame of each live stream, and the target template, the rendering unit is used to: determine the level value at any first position of the current video frame as the actual depth value at the first position, compare the actual depth value at the first position of the current video frame with the depth value at the corresponding position in the target template, and if the actual depth value at the first position is higher than or equal to the depth value at the corresponding position in the target template, the template test of the first position passes; determine the level value at any second position of the latest video frame as the actual depth value at the second position, compare the actual depth value at the second position of the latest video frame with the depth value at the corresponding position in the target template, and if the actual depth value at the second position is higher than or equal to the depth value at the corresponding position in the target template, the template test of the second position passes.

[0108] According to one or more embodiments of the present disclosure, when synthesizing the current video frame and the latest video frame based on the template test result, the rendering unit is used to: if the template test of any first position of the current video frame passes, render the first position of the current video frame; if the template test of any second position of the latest video frame passes, render the second position of the latest video frame; after completing the rendering of the positions that pass the template test in the current video frame and the latest video frame, obtain the synthesized target video frame.

[0109] According to one or more embodiments of the present disclosure, the level value of the latest video frame is higher than the level value of the current video frame.

[0110] According to one or more embodiments of the present disclosure, when determining the target template for template testing in the fragment shader rendering pipeline, the rendering unit is used to: set the depth value at all positions of the target template to the level value of the current video frame; and update the depth value at the corresponding area of ​​the latest video frame in the target template to the level value of the latest video frame according to the level value of the latest video frame and the preset layout information.

[0111] According to one or more embodiments of the present disclosure, before the rendering unit synthesizes the current video frame and the latest video frame of each live stream into a target video frame based on a single rendering view and according to preset layout information, the rendering unit is further used to: obtain the size and position of each view control in the preset list view control, use the size and position of any view control as the size and position of the second layer corresponding to the latest video frame in the target video frame, and obtain the preset layout information so that the second layer corresponding to the latest video frame is aligned with any view control in the preset list view control.

[0112] According to one or more embodiments of the present disclosure, when the rendering unit synthesizes the current video frame and the latest video frame into a target video frame based on a single rendering view and preset layout information, it is used to: synthesize the current video frame and the latest video frame into a target video frame based on a single rendering view and a single rendering thread according to preset layout information.

[0113] According to one or more embodiments of the present disclosure, when displaying the target video frame, the display unit is configured to: display the target video frame using a single surface view.

[0114] According to one or more embodiments of the present disclosure, when the rendering unit synthesizes the current video frame and the latest video frame into a target video frame based on a single rendering view and according to preset layout information, it is also used to: render the target video frame into a single frame buffer object; when the display unit displays the target video frame using a single surface view, it is used to: display the rendered target video frame cached in the frame buffer object using a single surface view.

[0115] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live broadcast processing method described in the first aspect and various possible designs of the first aspect.

[0116] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the live broadcast processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0117] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the live broadcast processing method as described in the first aspect and various possible designs of the first aspect is implemented.

[0118] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0119] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0120] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A live broadcast processing method, applied to a first live broadcast and microphone connection client, the method comprising: Receiving a live stream from a second live broadcast and microphone connection client, and caching the latest video frame of the live stream from the second live broadcast and microphone connection client; When acquiring the current video frame of the first live broadcast and microphone connection client, reading the cached latest video frame; Based on a single rendering view, synthesizing the current video frame and the latest video frame into a target video frame according to preset layout information; The target video frame is displayed.

2. The method according to claim 1, wherein: The synthesizing the current video frame and the latest video frame into a target video frame according to the preset layout information includes: Determine the size and position of the first layer corresponding to the current video frame and the second layer corresponding to the latest video frame in the target video frame according to the preset layout information; The current video frame is drawn into the first layer, the latest video frames are drawn into the second layers respectively, and the first layer and the second layer are synthesized into the target video frame.

3. The method according to claim 1, wherein: The synthesizing the current video frame and the latest video frame into a target video frame according to the preset layout information includes: Determine a target template for template testing in a fragment shader rendering pipeline, wherein a depth value at an area corresponding to the current video frame in the target template is a level value of the current video frame, and a depth value at an area corresponding to the latest video frame is a level value of the latest video frame; A template test is performed according to the level value of the current video frame, the level value of the latest video frame, and the target template, and the current video frame and the latest video frame are synthesized based on the template test result.

4. The method according to claim 3, wherein: The performing a template test according to the level value of the current video frame, the level value of the latest video frame and the target template includes: Determine the level value at any first position of the current video frame as the actual depth value at the first position of the current video frame, compare the actual depth value at the first position with the depth value at the corresponding position in the target template, and if the actual depth value at the first position is higher than or equal to the depth value at the corresponding position in the target template, the template test of the first position passes; The level value at any second position of the latest video frame is determined as the actual depth value at the second position of the latest video frame, and the actual depth value at the second position is compared with the depth value at the corresponding position in the target template. If the actual depth value at the second position is higher than or equal to the depth value at the corresponding position in the target template, the template test of the second position passes.

5. The method according to claim 4, wherein: The synthesizing the current video frame and the latest video frame based on the template test result includes: If the template test of any first position of the current video frame passes, rendering the first position of the current video frame; If the template test of any second position of the latest video frame passes, rendering the second position of the latest video frame; After rendering is completed at the positions that pass the template test in the current video frame and the latest video frame, the synthesized target video frame is obtained.

6. The method according to any one of claims 3 to 5, wherein: The level value of the latest video frame is higher than the level value of the current video frame.

7. The method according to claim 6, wherein: The step of determining a target template for template testing in a fragment shader rendering pipeline includes: Setting the depth values ​​at all positions of the target template to the level values ​​of the current video frame; According to the level value of the latest video frame and the preset layout information, the depth value at the area corresponding to the latest video frame in the target template is updated to the level value of the latest video frame.

8. The method according to claim 1, wherein: Before synthesizing the current video frame and the latest video frame into a target video frame based on a single rendering view according to preset layout information, the method further includes: Get the size and position of each view control in the preset list view control, use the size and position of any view control as the size and position of the second layer corresponding to the latest video frame in the target video frame, and obtain the preset layout information to align the second layer corresponding to the latest video frame with any view control in the preset list view control.

9. The method according to any one of claims 1 to 5, wherein: The synthesizing the current video frame and the latest video frame into a target video frame based on a single rendering view and according to preset layout information includes: Based on a single rendering view and a single rendering thread, the current video is rendered according to preset layout information. The frame and the latest video frame are synthesized into a target video frame.

10. The method according to claim 9, wherein: The displaying of the target video frame comprises: The target video frame is displayed using a single surface view.

11. The method according to claim 10, wherein: The method of synthesizing the current video frame and the latest video frame into a target video frame based on a single rendering view according to preset layout information further includes: Rendering the target video frame into a single frame buffer object; The adopting a single surface view to display the target video frame comprises: The rendered target video frames cached in the frame buffer object are displayed using a single surface view.

12. A live broadcast processing device, applied to a first live broadcast and microphone connection client, the device comprising: A receiving unit, configured to receive a live stream from a second live broadcast and microphone connection client; A cache unit, used to cache the latest video frame of the live stream of the second live broadcast and microphone connection client; When acquiring the current video frame of the first live broadcast and microphone connection client, reading the cached latest video frame; A rendering unit, configured to synthesize the current video frame and the latest video frame into a target video frame based on a single rendering view and according to preset layout information; A display unit is used to display the target video frame.

13. An electronic device comprising: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product, comprising computer executable instructions, and when a processor executes the computer executable instructions, the method according to any one of claims 1 to 11 is implemented.

Images