Live broadcast device and related live broadcast method

Abstract

The invention discloses a multi-platform live broadcast device and method. The device comprises a user interface service module and a streaming output service module. The user interface service module is used for controlling one or more image elements according to user interface setting so as to respectively determine the picture layout of the local monitoring image and the picture layout of the live broadcast output image. The streaming output service module is used for encoding the live broadcast output image to generate live broadcast streaming media, and streaming the live broadcast streaming media to a plurality of live broadcast platforms at the same time. Wherein the picture layout of the local monitoring image can be different from the picture layout of the live broadcast output image.

Description

Technical Field

[0001] The present invention relates to a live broadcast technology, in particular, to a live broadcast device with cross-platform synchronous live broadcast capability and independent control of screen layout and a related live broadcast method. Background Art

[0002] The advancement of network transmission and image coding technology has led to the vigorous development of live broadcast platforms. Today, live broadcasts of various styles and contents have become an important part of modern entertainment. In order to meet this trend, a variety of software and hardware devices that facilitate individual users to conduct live broadcasts have been launched on the market. These live broadcast devices enable users to conveniently transmit local video content to the live broadcast platform in the form of streaming media for users of the live broadcast platform to watch. However, there are several obvious limitations in existing live broadcast devices. First, existing live broadcast devices do not support multi-platform simultaneous live broadcasts, and live broadcast hosts cannot stream video content to different live broadcast platforms at the same time to achieve cross-platform simultaneous live broadcasts. In addition, existing live broadcast devices lack the function of dynamic bit rate adjustment. When the network connection status between the live broadcast device and the live broadcast platform changes, the bit rate of the streaming media cannot be automatically adjusted to ensure the viewing experience of the live broadcast viewers. Furthermore, existing live broadcast devices do not allow live broadcast hosts to adjust the screen layout during the live broadcast process; once the live broadcast activity begins, the screen layout will be fixed and cannot be adjusted in real time according to the needs of the live broadcast host or application scenarios. Finally, existing live broadcast equipment cannot perform differentiated processing on local monitoring image content and live broadcast output image content. These live broadcast equipment limit the image content of local monitoring image and live broadcast output image to be consistent, lacking adaptability to scenes or needs. As can be seen from the above, there are many areas in which existing live broadcast equipment needs to be improved. Summary of the invention

[0003] In view of this, one of the purposes of the present invention is to provide a live broadcast device and a live broadcast method. The live broadcast device and the live broadcast method have the following characteristics. First, the live broadcast device and the live broadcast method of the present invention can directly connect with multiple live broadcast platforms without using a third-party cloud service platform, and transmit the live streaming media to multiple live broadcast platforms at the same time, thereby realizing cross-platform synchronous live broadcast, and has the function of dynamically adjusting the bit rate of the live streaming media. Furthermore, the live broadcast device and the live broadcast method of the present invention support real-time screen layout adjustment. Even if the live broadcast activity starts and the live streaming media has been sent to the live broadcast platform, the user is still allowed to change the screen layout at any time. In addition, the live broadcast device and the live broadcast method of the present invention also support independent control of the screen layout, allowing users to use multiple screen layouts for local monitoring images and live broadcast output images.

[0004] An embodiment of the present invention provides a multi-platform live broadcast device, which includes: a user interface service module and a streaming output service module. The user interface service module is used to control one or more video elements according to user interface settings to respectively determine the screen layout of the local surveillance video and the screen layout of the live broadcast output video. The streaming output service module is used to encode the live broadcast output video to generate a live broadcast stream media, and stream the live broadcast stream media to multiple live broadcast platforms simultaneously. Among them, the screen layout of the local surveillance video can be different from the screen layout of the live broadcast output video.

[0005] An embodiment of the present invention provides a multi-platform live broadcast method, including: controlling one or more video elements according to user interface settings to respectively determine the screen layout of the local surveillance video and the screen layout of the live broadcast output video. Encoding the live broadcast output video to generate a live broadcast stream media, and streaming the live broadcast stream media to multiple live broadcast platforms simultaneously. Among them, the screen layout of the local surveillance video can be different from the screen layout of the live broadcast output video. Description of the Drawings

[0006] In order to further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the provided drawings are only for reference and illustration, and are not used to limit the present invention.

[0007] Figure 1A An application schematic diagram of the live broadcast device according to an embodiment of the present invention is shown.

[0008] Figure 1B A service layer architecture diagram of the live broadcast device according to an embodiment of the present invention is shown.

[0009] Figure 2A An illustration of how the user interface service module according to an embodiment of the present invention controls the screen layout is shown.

[0010] Figure 2B and Figure 2C A collaborative operation flowchart of the user interface service module and the streaming output service module according to an embodiment of the present invention is shown.

[0011] Figure 3 An architecture diagram of the user interface service module and the streaming output service module according to an embodiment of the present invention is shown.

[0012] Figure 4 A flowchart of the live broadcast method according to an embodiment of the present invention is shown.

[0013] Symbol Description 100 Live broadcast device 111, 112 Ports 113 Network interface 114 and 115 signal transmission interfaces 130 and 140 cameras 135 display device 145 video source 151, 152, and 153 remote playback devices 160 user interface service module 170 streaming output service module Steps S11 to S35, S110 to S120 310 status information 311 system UI 312 cursor 313 video source 314 and 315 camera sources 316, 317, 318, and 320 OSD handlers 319 PQ path 321 and 322 OSD mixing handlers 331 local output 332 encoder Detailed implementation

[0014] In the following, many specific details are described to provide the reader with an in - depth understanding of the embodiments of the present invention. However, those skilled in the art will understand how to implement the present invention in the absence of one or more specific details, or by using other methods, components, or materials, etc. In other cases, well - known structures, materials, or operations will not be shown or described in detail, so as to avoid obscuring the core concepts of the present invention.

[0015] The "embodiments" mentioned in the specification mean that the specific features, structures, or characteristics described in the embodiments may be included in at least one embodiment of the present invention. Therefore, the phrase "in one embodiment" that appears throughout this specification does not necessarily refer to the same embodiment. In addition, the aforementioned specific features, structures, or characteristics may be combined in any suitable form in one or more embodiments.

[0016] Reference Figure 1A, this figure is a schematic diagram of the application of the live broadcast device according to an embodiment of the present invention. As shown in the figure, the live broadcast device 100 can be used to generate live broadcast content for a live host. The live broadcast device 100 uses specific transmission protocols, such as: various types of Real-Time Messaging Protocols (RTMP) and various variants of RTMP (such as: RTMPS, RTMPE, RTMPT, RTMFP), or HTTPLive Streaming (HLS), or MPEG-DASH and other transmission protocols, so as to simultaneously transmit the streaming media including the live broadcast content to multiple live platforms, such as live platform A, live platform B, and live platform C. The streaming media is redistributed to different viewers' devices through live platform A, live platform B, and live platform C, such as remote playback device A (151), remote playback device B (152), and remote playback device C (153).

[0017] The live broadcast device 100 may include port A (111), port B (112), network interface 113, signal transmission interface A (114), and signal transmission interface B (115). Among them, port A (111) and port B (112) are respectively connected to the image acquisition devices 130 and 140 (such as cameras with dynamic image output functions), so as to use the acquired images output by cameras 130 and 140 as part of the live broadcast content. In one embodiment, port A (111) and port B (112) may be ports compliant with the Universal Serial Bus (USB) standard. The signal transmission interface A (114) can be connected to the display device 135 to display the local monitoring image generated by the live broadcast device 100 for the live host to view the current live broadcast content. Among them, the signal transmission interface A (114) may be a V-by-One interface. The signal transmission medium B (115) can be connected to the video source 145 to use the image output by the video source 145 as the basic image of the live broadcast content. Among them, the signal transmission interface B (115) may be a High Definition Multimedia Interface (HDMI) or a DisplayPort (DP). In one embodiment, the video source 145 may be a personal computer, mobile phone, tablet, game console, etc. The network interface 113 is connected to the Internet to encode the live broadcast output image generated locally by the live broadcast device 100, generate a streaming media, and send the streaming media to live platform A, live platform B, and live platform C via the Real-Time Messaging Protocol.

[0018] Figure 1B The service layer architecture diagram of the live broadcast device 100 in the embodiment of the present invention is shown. As shown in the figure, the live broadcast device 100 includes a user interface service module 160 and a streaming output service module 170. Among them, the function of the user interface service module 160 is (but not limited to): according to the user interface setting, control one or more video elements to respectively determine the screen layout of the local surveillance video and the screen layout of the live broadcast output video, so as to realize independent control of the screen layout of the local surveillance video and the live broadcast output video. Furthermore, the function of the streaming output service module 170 is (but not limited to): encode the live broadcast output video to generate a live broadcast streaming media, and stream the live broadcast streaming media to multiple live broadcast platforms at the same time, such as live broadcast platform A, live broadcast platform B, and live broadcast platform C. Among them, the streaming output service module 170 can learn the network connection status with live broadcast platform A, live broadcast platform B, and live broadcast platform C through the information reported by the network interface 113, so as to dynamically change the bit rate used when encoding the live broadcast output video.

[0019] Figure 2AIllustrates how the user interface service module 160 in an embodiment of the present invention controls the screen layout of the local monitoring image and the live output image. In this embodiment, the screen layout is composed of image elements 191 to 195. Among them, the image element 191 is used to display the timer user interface (UI), which shows the time elapsed during the current live event. The image element 192 is used to display the network connection status UI, which shows the network connection status between the current live device 100 and live platforms A, B, and C, such as: latency, connection status, and / or connection rate. The image elements 193 and 194 are respectively used to display the captured images output by the cameras 130 and 140. In one embodiment, when the image element 193 or the image element 194 displays the captured images output by the cameras 130 and 140, it can also display a Quick Response Code (QR code) at the same time, so as to provide information about the current live event (such as: website address) to the live viewers. The bottommost image element 195 can be the basic image output by the video source 145 as the basic content of the live broadcast. It is worth mentioning that the display position of the QR code can be set not only at the four corners of the image elements 193 and 194, but also at any position in the screen layout. For example, it can be set at the top of the screen layout like the image elements 191 and 192, such as QR_A, or at the bottom of the screen layout, such as QR_B, or at any corner of the screen layout, such as QR_C, or at any position in the center of the screen layout, such as QR_D. Furthermore, the user interface service module 160 can independently control the transparency, position, size, and / or Z-axis position (i.e., the stacking order or occlusion relationship with other image elements) of each image element 191 to 195 in the local monitoring image and the live output image according to the user interface setting data, so that the local monitoring image and the live output image can have different screen layouts.

[0020] It is worth mentioning that the types and quantities of the image elements 191 to 195 mentioned in the foregoing embodiments are not limitations of the present invention. In other embodiments of the present invention, the screen layout of the local monitoring image and the live output image may include more or fewer image elements, and image elements different from Figure 2A those shown in the illustrated embodiment. For example, the image element can also display a status information UI other than the timer UI and the network connection status UI, or the cursor or pointer of a controller (such as a mouse). In addition, the sizes and positions of these image elements may also be Figure 2A different from those of the image elements in the illustrated embodiment.

[0021] Figure 2B andFigure 2C Further shown is a flowchart of the collaborative operation of the user interface service module 160 and the streaming output service module 170 in an embodiment of the present invention. First, in step S11, the live application corresponding to the live device 100 is started. In step S12, the user interface setting data is loaded from the database of the live device 100. In step S13, the editing mode provided by the user interface service module 160 is entered, where the editing mode is used to edit the screen layout of the local monitoring image and the live output image. In step S14, it is determined whether there is user interface setting data stored in the database; if not, the process proceeds to step S15; if so, the process proceeds to step S17. In step S15, the image elements for displaying the timer UI are initialized, and then in step S16, the image elements including the timer UI are displayed. On the other hand, if there is user interface setting data stored in the database, then in step S17, the stored information UI is restored according to the user interface setting data, and in step S18, the stored information UI is displayed. Note that the information UI mentioned here and below can generally refer to the combination of various UIs displayed in any of the image elements in the aforementioned screen layout, including but not limited to: the timer UI, the network connection status UI, the status information UI, the cursor or pointer of the controller, the captured images output by the cameras 130 and 140, and the base image output by the video source 145.

[0022] In step S19, it is determined whether the information UI has been started; if not, the process proceeds to step S20, and if so, the process proceeds to step S21. In step S20, since the information UI is closed, the information UI can be started in step U23. On the other hand, in step S21, since the information UI has been started, it is possible to either proceed to the next step of the process or enter step S22 to close the information UI. In the state where the information UI is started, it is possible to enter steps S24, S25, or S26, and according to the needs and operations of the live host, move (position adjustment), scale (size adjustment), or Z-axis position (stacking order / occlusion relationship) of individual image elements in the information UI can be performed respectively. After the adjustment of the information UI is completed, the relevant parameters will be stored in the database as new user interface setting data or updated existing user interface setting data (step S27). In steps S28 and S29, the editing mode is exited and the live mode is entered. In step S30, the real-time information service provided by the user interface service 160 is started (i.e., the information UI is displayed in the local monitoring image or the stream output image). In step S31, the previously set user interface setting data is loaded from the database. In step S32, according to the loaded user interface setting data, the composition and display settings of the information UI are updated. In step S33, the streaming service provided by the streaming output service module 170 is started, the local monitoring image is output to the local display device 135, and the streaming output image is encoded into a live streaming media and output to the live streaming platform. In step S34, the composition and display of the information UI are updated. In this step, since the network connection status UI (if any) in the information UI needs to display the network connection status information in real time, and the live host may also adjust the information UI during the live broadcast (for example, blocking specific image elements, or adjusting the transparency, size, position, and Z-axis position of specific image elements), therefore, after the live event starts, it is still necessary to continuously update the composition and display settings of the information UI. Finally, in step S35, according to the updated composition and display settings of the information UI, the information UI is displayed in the local monitoring image or the stream output image.

[0023] In the present invention, the user interface service module 160 can make the screen layout of the local monitoring image different from the screen layout of the live output image. Therefore, it needs to be implemented through an architecture that can independently control image elements. Please refer to Figure 3 for the description. As Figure 3The architecture diagrams of the user interface service module 160 and the streaming output service module 170 shown. Among them, the on-screen display (OSD) handlers 316, 317, 318, and 320 are respectively used to control the display or masking of different video elements. The OSD mixing handlers 321 and 322 are used to control the mixing of the video elements displayed by the OSD handlers 316, 317, 318, and 320, so as to generate the local surveillance video and the live output video respectively. Furthermore, the local surveillance video is provided by the local output 331 to the signal transmission interface A (114) as shown in Figure 1A and transmitted to the display device 135 for display. The live output video is encoded by the encoder 332 to generate a streaming media, and then provided to the network interface 113 as shown in Figure 1A and transmitted to the live platform.

[0024] Furthermore, the status information 310 provides the transparency information, size information, position information, and substantial content information of various status information UIs to be displayed (including: timer UI, network connection status UI) to the OSD handler 2 (316) to draw relevant images. The system UI 311 provides the UI information of the underlying operating system to the OSD handler 1 (317) to draw relevant images. The cursor 312 provides the position information of the cursor to the OSD handler 3 (318) to draw relevant images. The PQ path 319 obtains the base video from the video source 313. The OSD handler 4 (320) obtains the captured video from the camera sources 314 and 315. Furthermore, the OSD mixing handlers 321 and 322 mix the images drawn or the videos output by the OSD handlers 316, 317, 318, and 320 in the correct stacking order or occlusion relationship according to the Z-axis position corresponding to each video element, so as to draw the local surveillance video and the live output video. Among them, the OSD mixing handlers 321 and 322 can also determine whether to display or mask the images drawn by one or more of the OSD handlers 316, 317, 318, and 320. Therefore, the local surveillance video and the live output video can have different screen layouts. For example, the live host can control to display a specific video element in the local surveillance video but not in the live output video, and vice versa. In addition, the live host can automatically or manually mask a specific video element that was originally displayed in the local surveillance video or the live output video during the live event, so as to change the screen layout in real time. Note that in different embodiments of the present invention, the OSD handler may be used to draw or output more than one video element, and the OSD handlers processed by each OSD mixing handler may also be different from Figure 3 the example in.

[0025] Based on the above content, Figure 4 A simplified process of the live broadcast method in an embodiment of the present invention is shown. As shown in the figure, the simplified process includes the following steps:

[0026] Step S110: According to the user interface setting, control one or more video elements to respectively determine the screen layout of the local surveillance video and the screen layout of the live broadcast output video; and

[0027] Step S120: Encode the live broadcast output video to generate a live streaming media stream, and stream the live streaming media stream to multiple live broadcast platforms simultaneously, wherein the screen layout of the local surveillance video may be different from the screen layout of the live broadcast output video.

[0028] Since the principles and specific details of the above steps have been described in detail through previous embodiments, they will not be elaborated here. It should be noted that the above process can be better implemented to generate and stream the live broadcast output video by adding other additional steps, or making appropriate modifications and adjustments, so as to obtain a better live broadcast and relay experience.

[0029] Embodiments of the present invention can be specifically implemented as devices, methods, or computer program products. Accordingly, embodiments of the present invention can take the form of an entity implemented entirely by hardware, an entity implemented entirely by software (including firmware, resident software, microcode, etc.), or an entity combining software and hardware aspects, which can be generally referred to as "modules" or "systems". In addition, embodiments of the present invention can take the form of a computer program product embodied in any tangible expression medium, and the medium has computer-usable program code embodied in the medium. In terms of hardware, the present invention can be implemented by applying any of the following technologies or related combinations: the individual operation logic of logic gates capable of performing logical functions according to data signals, and application specific integrated circuits (ASICs), programmable gate arrays (PGAs), or field programmable gate arrays (FPGAs) with appropriate combined logic.

[0030] Flowcharts and block diagrams illustrate the architecture, functionality, and operation of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of program code, which includes one or more executable instructions to implement a particular logical function. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a special-purpose hardware-based system, or a combination of special hardware and computer program instructions. These computer program instructions may be stored in a readable computer medium to direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the readable computer medium cause the production of an article including implementation of the functions / operations specified in the block or combination of blocks of the flowchart and / or block diagram.

[0031] Although the embodiments of the present invention are described as above, these embodiments are not intended to limit the present invention. Those of ordinary skill in the art may make variations to the technical features of the present invention based on the content explicitly or implicitly disclosed in the present invention. All such variations fall within the scope of patent protection sought by the present invention. In other words, the scope of patent protection of the present invention shall be defined by the scope defined in the claims of the present invention application.

Claims

1. A multi-platform live broadcast device, characterized in that, the multi-platform live broadcast device includes: a user interface service module, configured to control one or more video elements according to user interface settings, so as to respectively determine the screen layout of the local surveillance video and the screen layout of the live broadcast output video; and a streaming output service module, configured to encode the live broadcast output video to generate a live broadcast streaming media, and stream the live broadcast streaming media to multiple live broadcast platforms simultaneously; wherein, the screen layout of the local surveillance video may be different from the screen layout of the live broadcast output video.

2. The live broadcast device according to claim 1, characterized in that, during the process that the streaming output service module streams the live broadcast streaming media to the multiple live broadcast platforms, the user interface service module may adjust the screen layout of the live broadcast output video according to the modified user interface settings.

3. The live broadcast device according to claim 1, characterized in that, the user interface service module is configured to determine and control the transparency, position, size, and / or Z-axis position of the one or more video elements according to the user interface settings.

4. The live broadcast device according to claim 1, characterized in that, the one or more video elements are used to display one of a timer user interface, a network connection status user interface, a status information user interface, a cursor or pointer of a controller, a captured video output by a video capture device, and a base video output by a video source.

5. The live broadcast device according to claim 4, characterized in that, the one or more video elements are used to display the captured video and a quick response code related to the live broadcast activity.

6. The live broadcast device according to claim 6, characterized in that, the live broadcast device further includes: a plurality of on-screen display (OSD) processing modules, respectively configured to draw images corresponding to different video elements; a first OSD mixing processing module, configured to generate the local surveillance video according to the images drawn by a part or all of the plurality of OSD processing modules; and a second OSD mixing processing module, configured to generate the live broadcast output video according to the images drawn by a part or all of the plurality of OSD processing modules.

7. The live broadcast device according to claim 1, characterized in that, the streaming output service module can be used to change the bit rate corresponding to the live broadcast streaming media according to the network connection status between the live broadcast device and one of the multiple live broadcast platforms.

8. A multi-platform live broadcast method, characterized in that, the multi-platform live broadcast method includes: controlling one or more video elements according to user interface settings, so as to respectively determine the screen layout of the local surveillance video and the screen layout of the live broadcast output video; and encoding the live broadcast output video to generate a live broadcast streaming media, and streaming the live broadcast streaming media to multiple live broadcast platforms simultaneously; wherein, the screen layout of the local surveillance video may be different from the screen layout of the live broadcast output video.

9. The live broadcast method according to claim 8, characterized in that, The multi-platform live streaming method further includes: modifying the user interface settings; and during the process of streaming the live streaming media stream to the multiple live streaming platforms, adjusting the screen layout of the live streaming output image.

10. The live streaming method according to claim 8, wherein, the step of controlling the one or more image elements according to the user interface settings includes: determining and controlling the transparency, position, size, and / or Z-axis position of the one or more image elements according to the user interface settings.

Images