Video stream split-screen display method, device, system, stream pulling device and storage medium

Abstract

Embodiments of the present application disclose a video stream split-screen display method, device, system, streaming device and storage medium. The streaming device obtains a video stream pushed by a push streaming client on a streaming media server through a WebRTC protocol in a local browser, converts the video stream into a WebRTC video stream, and then plays the WebRTC video stream in real time by using multiple VIDEO tags in HTML5. The technical solution of the present application combines the WebRTC protocol with a traditional push-pull streaming media system based on the ultra-low delay characteristic of the WebRTC protocol, and achieves the technical purpose of playing a video stream on multiple screens within a delay of 35-70 ms in a scene where the streaming media server relays the push streaming video stream. Thus, the delay is reduced, and the user experience during real-time multi-screen video playing is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of streaming media and network transmission, and in particular to a video stream split-screen display method, device, system, stream pulling device and storage medium. BACKGROUND

[0002] In recent years, with the progress of science and technology, after the File and Web eras, we have now entered the era of "Flows / Streams", and the pace of life of the public has become faster and faster in various information flows, data flows and media flows.

[0003] The most typical application of the transition to "Flows" is the Streaming Midea technology, which refers to the use of streaming technology on the network to publish audio, video and other multimedia files. Streaming refers to the encoding and compression of audio and video multimedia files into small compressed packages through a certain algorithm, and the streaming media server transmits continuously and in real time through a specific network protocol. The user end receives the compressed package and plays it back by real-time decompression through the playback software.

[0004] There are many application scenarios in the field of streaming media technology at present, including video interaction, remote education and medical treatment, live streaming, monitoring, etc. In some scenarios, such as monitoring, it is necessary to display multiple camera monitored screens on the web page. In the prior art, when the push stream end device (such as a monitoring camera, a mobile device, etc.) pushes the collected video data to the streaming media server, one of the processing methods is to convert the video stream into a video stream in the HLS (HTTP Live Streaming, i.e., HTTP-based adaptive bitrate streaming protocol), MSE (Media Source Extensions, i.e., Media Source Extensions) or FLV (Flash Video) protocol format that can be played directly in the web browser, and then the web page pulls and plays the converted video stream. This scheme has the following problems:

[0005] 1. HLS, a streaming media network transmission protocol proposed by Apple, is primarily used for audio and video services on PCs and Apple devices. Because HLS uses the HTTP protocol for data transmission, it generally doesn't encounter firewall blocking. Furthermore, HLS itself implements adaptive bitrate, allowing devices with different bandwidths to automatically switch to the video playback at the most suitable bitrate. However, HLS plays video in segments, including an M3U8 index file, TS media segment files, and a key encryption string file. Due to its segmented loading characteristic, the playback latency is relatively long. For example, if an M3U8 file contains five TS files, each with a playback duration of 2 seconds, then the playback duration of one M3U8 file is 10 seconds. This means that the playback progress of this M3U8 file is at least 10 seconds in progress. Although Apple released a new solution at WWDC 2019 that can reduce the latency to 1-2 seconds, this remains a significant drawback for scenarios with high real-time requirements.

[0006] 2. Regarding MSE, it's a new Web API supported by major browsers, providing the ability to implement plugin-free, web-based streaming media. Using MSE, users can dynamically modify media streams without any plugins. However, if some browsers don't support MSE, creating MSE-compatible versions for those browsers is very time-consuming and laborious, consuming significant computer resources and energy. Additionally, the video playback latency is around 3 seconds, which is slightly high.

[0007] 3. As for the FLV protocol, FLV is a streaming media format that is widely supported due to its small size and relatively simple protocol. It has the advantages of high maturity and no pressure on high concurrency. However, its disadvantages are that it requires the integration of an SDK to play, and it is mainly affected by the size of the GOP and the backlog of TCP weak network transmission, resulting in a relatively high playback delay, generally between 2 and 3 seconds.

[0008] Therefore, although these protocols each have their own advantages, they all inevitably suffer from high video playback latency. High latency leads to a series of problems caused by untimely video updates, which directly reduces the user experience.

[0009] Given the problems with live video streaming on web pages in the existing technologies mentioned above, there is an urgent need to propose a video stream split-screen display method that can reduce latency and improve the user experience when playing videos in real time on multiple screens. Summary of the Invention

[0010] Therefore, the embodiments of the present application provide a video stream split screen display method, device, equipment and system and storage medium, and a streaming device obtains a video stream pushed by a push streaming client on a streaming media server through a WebRTC protocol in a local browser, converts the video stream into a WebRTC video stream, and then plays the WebRTC video stream in real time through multiple VIDEO tags in HTML5. The technical solution of the present application combines the WebRTC protocol with a traditional push-pull streaming media system based on the ultra-low delay characteristic of the WebRTC protocol, and achieves the technical purpose of playing a video stream on multiple screens within a delay of 35-70 ms in a scenario where a streaming media server is needed to relay a push video stream. Thus, the delay is reduced, and the user experience of real-time multi-screen video playing is improved.

[0011] In a first aspect, the embodiments of the present application provide a video stream split screen display method applied to a streaming device, wherein the streaming device is installed with a browser supporting a Web Real-Time Communication (WebRTC) protocol, and the method comprises the following steps.

[0012] Preforming a streaming configuration locally in advance to generate a stream list, wherein the stream list comprises one or more stream list items, and each stream list item comprises a streaming address of a corresponding video stream in a streaming media server;

[0013] In response to a trigger instruction for playing a video stream corresponding to a stream list item in the stream list, obtaining a streaming address of the video stream in the streaming media server, and pulling the video stream from the streaming media server based on the streaming address of the video stream in the streaming media server, wherein the video stream is an original RTSP video stream sent by a push streaming client to the streaming media server, or an RTSP video stream transcoded by the streaming media server;

[0014] Decoding and unpacking the video stream to generate a WebRTC video stream conforming to the WebRTC protocol;

[0015] Split screen displaying the WebRTC video stream in a split screen interface generated according to a preset split screen format and based on a Video tag of a local browser HTML5.

[0016] Preferably, the preformed streaming configuration locally in advance to generate a stream list specifically comprises the following steps.

[0017] Obtaining stream configuration data of each video stream input by a user to generate a corresponding stream list item;

[0018] Generating the stream list based on the stream list items of each video stream;

[0019] The stream configuration data comprises a video stream name and a streaming address.

[0020] Preferably, the stream configuration data further comprises: a stream playing type; the stream playing type comprises: continuous playing and click playing.

[0021] Preferably, the pulling the video stream from the streaming media server based on the pull stream address of the video stream corresponding to the stream list item in the streaming media server comprises:

[0022] establishing a WebRTC connection channel based on a WebRTC protocol interface in the local browser and the streaming media server, and pulling the video stream through the WebRTC connection channel.

[0023] Preferably, the splitting and displaying the WebRTC video stream in the split screen interface generated according to the preset split screen format and based on the Video tag of the local browser HTML5 comprises:

[0024] splitting the split screen interface into a plurality of independent display windows according to the preset split screen format based on the Video tag of the local browser HTML5, each display window corresponding to a Video for playing a WebRTC video stream.

[0025] Preferably, each display window contains an area for responding to the addition and playing of a video stream, for receiving a trigger instruction of a user for adding and playing a video stream corresponding to a stream list item in the stream list in the display window.

[0026] In a second aspect, the embodiments of the present application further provide a video stream split screen display device, which is arranged in a pull stream equipment, the pull stream equipment is installed with a browser supporting Web Real-Time Communication (WebRTC) protocol, and the device comprises:

[0027] a stream list generation module arranged to generate a stream list by performing pull stream configuration in advance locally; wherein the stream list contains one or more stream list items, and each stream list item contains a pull stream address of a video stream corresponding to the stream list item in a streaming media server;

[0028] a pull stream module arranged to, in response to a trigger instruction for playing a video stream corresponding to a stream list item in the stream list, acquire a pull stream address of the video stream corresponding to the stream list item in the streaming media server, and pull the video stream from the streaming media server based on the pull stream address of the video stream corresponding to the stream list item in the streaming media server, wherein the RTSP video stream is an original RTSP video stream sent to the streaming media server by a push stream client, or an RTSP video stream after transcoding by the streaming media server;

[0029] The stream decoding module is configured to decode and unpack the video stream to generate a WebRTC video stream conforming to a WebRTC protocol.

[0030] The stream splitting display module is configured to split and display the WebRTC video stream in a split screen interface generated according to a preset split screen format and based on a local browser HTML5 Video tag.

[0031] In a third aspect, an embodiment of the present application provides a stream pulling device, comprising a memory and a processor, wherein the processor executes program instructions in the memory to implement the method in the first aspect.

[0032] In a fourth aspect, an embodiment of the present application provides a video stream splitting display system, comprising a stream pushing client, a stream media server and the stream pulling device in the third aspect.

[0033] In a fifth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium is configured to store a computer program, and the computer program is configured to implement the method in the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0034] The above and other objects, features and advantages of the present application will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0035] Figure 1 is a flow chart of a video stream splitting display method according to an embodiment of the present application;

[0036] Figure 2 is a structural schematic diagram of a video stream splitting display apparatus according to an embodiment of the present application;

[0037] Figure 3 is a hardware structural schematic diagram of a stream pulling device according to an embodiment of the present application;

[0038] Figure 4 is a structural schematic diagram of a video stream splitting display system according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present application is described below based on embodiments, but the present application is not limited to only these embodiments. In the following detailed description of the present application, some specific details are described in detail. The present application can also be understood without the description of these details by those skilled in the art. In order to avoid obscuring the essence of the present application, well-known methods, processes, flows, elements and circuits are not described in detail.

[0040] In addition, those of ordinary skill in the art should understand that the drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0041] Meanwhile, it should be understood that in the following description, "circuitry" refers to an electrical line, which is formed by at least one element or sub-circuit through electrical or electromagnetic connection. When an element or circuit is said to be "connected to" another element or said to be "connected between" two nodes, it can be directly coupled or connected to another element or there can be an intermediate element, the connection between the elements can be physical, logical, or a combination thereof. On the contrary, when an element is said to be "directly coupled to" or "directly connected to" another element, it means that there is no intermediate element between the two.

[0042] Unless the context clearly requires otherwise, throughout the description, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to".

[0043] In the description of the present application, it should be understood that the terms "first", "second" and the like are used only for descriptive purposes and should not be construed as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0044] The technical scheme of the present application adopts a browser / server (B / S) architecture deployment, and is divided into a pull stream device, a streaming media server and a push stream client. The pull stream device can be a PC or a mobile terminal, and is installed with a browser supporting a WebRTC (Web Real-Time Communication) protocol, such as a Google Chrome 23, a Mozilla Firefox 22 and an Opera 18 browser suitable for a PC, or a Google Chrome 28 (starting from version 29, it is enabled by default), a Mozilla Firefox 24 and an Opera Mobile 12 browser suitable for an Android client, and directly plays a video stream through a browser webpage interface. The streaming media server is deployed through a NodeServerProject based on Node.js, supports RTSP, RTMP / RTMPE / RTMPT / RTMPTE and HTTP, and only supports video data in an H.264 format and audio data in an AAC format. The Node.js uses an event-driven, non-blocking I / O model, and is a JavaScript runtime environment based on a Chrome V8 engine. The push stream client can be an Android client or a network monitoring camera (such as a Hikvision camera) with a push stream function. For the Android client, a push stream program package (such as an rtmp-rtsp-stream-client-java) is installed to transmit audio and video streams to the streaming media server in an RTMP or RTSP protocol.

[0045] The technical scheme of the present application relates to a method for splitting and displaying a video stream, which is applied to a stream-pulling device, and the stream-pulling device is installed with a browser supporting Web Real-Time Communication (WebRTC) protocol. The method comprises the following steps: S101, obtaining a video stream pushed by a stream-pushing client on a stream media server through the WebRTC protocol in the local browser; S102, converting the video stream into a WebRTC video stream; and S103, playing the WebRTC video stream in real time through multiple VIDEO tags in HTML5. The WebRTC protocol is a set of APIs supporting real-time audio and video in a web browser. In addition, it is an open source technology for real-time communication in a web page without any plug-in installed in the browser. It directly adds real-time media communication between the browser and the device, and realizes direct connection between the client and the client without storing and forwarding related real-time information through a server. Currently, it is mainly applied to video conferencing and microphone connection scenes. It uses a UDP private protocol to push the media stream, without creating discrete media segments. It is connectionless, without handshaking confirmation and connection closing mechanism when the TCP connection is disconnected. Based on the above two points, the WebRTC can achieve millisecond-level ultra-low delay, which is much lower than the delay of CDN distribution based on the RTMP protocol. The present application combines the ultra-low delay of the WebRTC protocol with the traditional push-pull stream media system, and realizes the technical purpose of playing the video stream in multiple screens within 35-70 ms delay in the scene of needing the stream media server to transfer the pushed video stream.

[0046] Figure 1 The present application is a method for splitting and displaying a video stream, which is applied to a stream-pulling device, and the stream-pulling device is installed with a browser supporting Web Real-Time Communication (WebRTC) protocol. The method comprises the following steps: S101, obtaining a video stream pushed by a stream-pushing client on a stream media server through the WebRTC protocol in the local browser; S102, converting the video stream into a WebRTC video stream; and S103, playing the WebRTC video stream in real time through multiple VIDEO tags in HTML5. The WebRTC protocol is a set of APIs supporting real-time audio and video in a web browser. In addition, it is an open source technology for real-time communication in a web page without any plug-in installed in the browser. It directly adds real-time media communication between the browser and the device, and realizes direct connection between the client and the client without storing and forwarding related real-time information through a server. Currently, it is mainly applied to video conferencing and microphone connection scenes. It uses a UDP private protocol to push the media stream, without creating discrete media segments. It is connectionless, without handshaking confirmation and connection closing mechanism when the TCP connection is disconnected. Based on the above two points, the WebRTC can achieve millisecond-level ultra-low delay, which is much lower than the delay of CDN distribution based on the RTMP protocol. The present application combines the ultra-low delay of the WebRTC protocol with the traditional push-pull stream media system, and realizes the technical purpose of playing the video stream in multiple screens within 35-70 ms delay in the scene of needing the stream media server to transfer the pushed video stream. Figure 1

[0047] Step S110: performing stream-pulling configuration in advance, and generating a stream list; wherein the stream list comprises one or more stream list items, and each stream list item comprises a stream-pulling address of a corresponding video stream in the stream media server.

[0048] In this step, the related information of the video stream to be played in the stream-pulling device is configured, and a stream list item is formed for summarizing each video stream, so that when a certain video stream needs to be played, the user can select an item in the interface for real-time playing and displaying.

[0049] In some exemplary embodiments, a trigger menu item or a list item for performing stream-pulling configuration and stream list display can be set in an interface, and the stream-pulling configuration and the stream list display are started by receiving a trigger operation of the user.

[0050] ​Specifically, after receiving a trigger operation of a user to perform the pull streaming configuration, a pull streaming configuration interface is entered, in which information items for configuring the video streams to be pulled from the streaming media server are included, and each video stream is configured by obtaining the stream configuration data of each information item of each video stream input by the user, wherein the stream configuration data includes but is not limited to: a video stream name and a pull streaming address. In some exemplary embodiments, the video stream name can be an arbitrary string, such as: my room or 123; and the format of the pull streaming address is: rtsp: / / <ip> : <port> / path, where ip is the address of the streaming server, port is the port of the streaming server, and path is a custom configuration parameter. In some example embodiments, the user learns the address of the streaming server and the port in advance, and then customizes the path parameter to distinguish different push streaming clients, and then sets the push streaming address in the push streaming client. If the push streaming protocol uses the RTMP protocol, the push streaming address is an address prefixed with rtmp: / / , then after the push streaming client pushes the video stream to the streaming server, the streaming server transcodes the RTMP video stream into an RTSP video stream, and the pull streaming address of the transcoded RTSP video stream is changed to an address prefixed with rtsp: / / , for example: if the push streaming address is rtmp: / / 124.139.232.61:1935 / live / 123, then the pull streaming address is rtsp: / / 124.139.232.61:1935 / live / 123; if the push streaming protocol uses the RTSP protocol, the push streaming address is an address prefixed with rtsp: / / , then after the push streaming client pushes the video stream to the streaming server, the streaming server only acts as a transit center for the video stream and does not need to perform transcoding processing, so the push streaming address is the pull streaming address. Thus, the user can learn the pull streaming address in advance under the condition of knowing the push streaming address, and then configure the pull streaming address information item in the pull streaming configuration interface.

[0051] In some example embodiments, the stream configuration data further includes a stream playing type, and the stream playing type includes continuous playing and click playing. The continuous playing means that the streaming server has continuously obtained the video stream from the push streaming client before the pull streaming device plays the video stream; the click playing means that the streaming server starts to obtain the video stream from the push streaming client only when the pull streaming device plays the video stream.

[0052] After all the information items of the video stream are configured in the pull streaming configuration interface, the save operation is clicked, and a stream list item of the video stream is generated. Next, if other video streams need to be configured, the pull streaming configuration interface is re-entered to configure them. Then, the stream list items of the video streams are aggregated into a table to generate a stream list. In some example embodiments, the stream list display menu item or list item set in the trigger interface can be used to view all the set stream list items.

[0053] Step S120: in response to a trigger instruction for playing a video stream corresponding to a stream list item in the stream list, obtaining a pull stream address of the video stream corresponding to the stream list item in the stream media server, and pulling the video stream from the stream media server based on the pull stream address of the video stream corresponding to the stream list item in the stream media server, wherein the video stream is an original RTSP video stream sent by the push stream client to the stream media server, or an RTSP video stream after transcoding by the stream media server.

[0054] Specifically, in the step S110 of triggering the playing of the video stream configured by the pull stream device, the specific triggering manner can be determined according to the specific project requirement. In some exemplary embodiments, a menu item or a list item for triggering the split screen display interface can be added in the interface for displaying the pull stream configuration and the stream list display. After clicking the menu item or the list item for triggering the split screen display interface, the real-time playing and display of all or part of the configured video streams are triggered, or in the display area for displaying a specific video stream in the split screen display interface, the real-time playing of a video stream in the stream list item is triggered by clicking the add and play video stream button set in the display area.

[0055] After the pull stream device receives the trigger instruction for playing the video stream corresponding to the stream list item in the stream list, the trigger instruction is responded to, and the specific operation is as follows:

[0056] Firstly, the pull stream address of the video stream corresponding to the stream list item in the stream media server is obtained, which can be achieved by calling a related API function interface in a specific example, wherein the pull stream address is the RTSP protocol format pull stream address input in the pull stream address information item in the pull stream configuration interface.

[0057] Then, the video stream is pulled from the stream media server based on the pull stream address of the video stream corresponding to the stream list item in the stream media server, wherein the video stream is an original RTSP video stream sent by the push stream client to the stream media server, i.e. the RTSP video stream pushed by the push stream client to the stream media server in the RTSP protocol, or the RTSP video stream after transcoding by the stream media server from the RTMP protocol pushed by the push stream client to the stream media server.

[0058] Specifically, when the video stream corresponding to the stream list item is pulled from the streaming media server based on the pull stream address of the video stream in the streaming media server, then a WebRTC connection channel is established between the local browser and the streaming media server based on the WebRTC protocol interface in the local browser, and the video stream is pulled through the WebRTC connection channel. Specifically, the RTCPeerConnection interface of the WebRTC protocol needs to be called to establish the WebRTC connection channel between the local (pull stream device) and the remote (streaming media server), and the interface provides the implementation of the methods for creating, maintaining, monitoring, and closing the connection. In a specific implementation, the local related parameters need to be initialized first, and then a local RTCPeerConnection object is created, and then: 1. Exchange ICE, that is, the WebRTC uses the ICE (Interactive Connectivity Establishment) framework protocol, and the local and remote need to exchange network information with each other before establishing the connection, and inform each other's address; 2. Exchange SDP, that is, the WebRTC uses the SDP (Session Description Protocol) to exchange the media information of the local and remote, such as resolution format and other information, mainly to exchange the offer (Offer) and answer (Answer) of the two parties; 3. The remote adds the video stream to be transmitted and transmits it to the local.

[0059] In some example embodiments, the specific implementation code is as follows:

[0060]

[0061]

[0062]

[0063]

[0064] Step S130: decoding and unpacking the video stream to generate a WebRTC video stream conforming to the WebRTC protocol.

[0065] Since the WebRTC protocol currently only supports H.264 format video data, the obtained video stream needs to be decoded and unpacked before playing to generate a WebRTC video stream conforming to the WebRTC protocol, that is, an H.264 video stream.

[0066] Step S140: splitting and displaying the WebRTC video stream in a split screen interface generated according to a preset split screen format and based on the Video tag of the local browser HTML5.

[0067] The "preset split-screen format" referred to in this step refers to the definition of the format of splitting a display interface into rows and columns when splitting a video stream for display, such as a 2*2 split-screen format, a 4*4 split-screen format, and a 7*7 split-screen format. If it is a 2*2 split-screen format, one display interface displays four videos. If it is a 4*4 split-screen format, one display interface displays 16 videos. If it is a 7*7 split-screen format, one display interface displays 49 videos. Other split-screen formats can also be defined according to actual needs. In some example embodiments, the default split-screen format can be set to a 4*4 split-screen format, and a menu item for selecting different split-screen formats can be set in the split-screen interface.

[0068] After the split-screen format is determined, the WebRTC video streams are split-screen displayed in the split-screen interface generated based on the local browser HTML5 Video tag. Specifically, based on the local browser HTML5 Video tag, the split-screen interface is divided into multiple independent display windows according to the preset split-screen format, and each display window corresponds to a Video for playing a WebRTC video stream. Each display window includes an area for responding to the addition and playing of a video stream, which is used to receive a trigger instruction from a user for adding and playing a video stream corresponding to a stream list item in the stream list in the display window.

[0069] In some example embodiments, in the interface provided with the menu item or list item of the split-screen display interface, after clicking the menu item or list item of the split-screen display interface, the split-screen interface appears. After entering the split-screen interface, no video is played by default. After clicking the area for adding a video stream in one of the display windows, and then selecting one of the stream list items in the pop-up stream list, the video stream corresponding to the stream list item is added and played in the corresponding display window after sequentially performing steps S120-S130 and this step. If other video streams need to be added, the operation can be repeated. Each display window can repeatedly play the same video stream or different video streams. Thus, the purpose of playing multiple video streams in real time in one display interface is achieved.

[0070] From the above steps, the stream pulling device in the embodiment of the application obtains the video stream pushed by the push stream client on the stream media server through the WebRTC protocol in the local browser, and converts the video stream into a WebRTC video stream, and then utilizes multiple VIDEO tags in HTML5 to split the screen and play the WebRTC video stream in real time. The technical scheme of the application is based on the ultra-low delay characteristic of the WebRTC protocol, combines the WebRTC protocol with the traditional push-pull stream media system, and realizes the technical purpose of playing the video stream on multiple screens within 35-70 ms in the scene of needing the stream media server to transfer the push stream video stream. Thus, the delay is reduced, and the user experience during real-time playing of the video is improved.

[0071] Figure 2 is a structural schematic diagram of a video stream split screen display device of the embodiment of the application, which is arranged in a stream pulling device, the stream pulling device is installed with a browser supporting the Web Real-Time Communication WebRTC protocol, and comprises the following modules:

[0072] The stream list generation module 210 is arranged to generate a stream list by performing stream pulling configuration locally in advance, wherein the stream list comprises one or more stream list items, and each stream list item comprises a pulling address of a corresponding video stream in a stream media server;

[0073] The stream pulling module 220 is arranged to obtain the pulling address of the video stream corresponding to the stream list item in the stream media server in response to a trigger instruction for playing the video stream corresponding to the stream list item in the stream list, and pull the video stream from the stream media server based on the pulling address of the video stream corresponding to the stream list item in the stream media server, wherein the RTSP video stream is an original RTSP video stream sent by a push stream client to the stream media server, or an RTSP video stream after transcoding by the stream media server;

[0074] The stream decoding module 230 is arranged to decode and unpack the video stream to generate a WebRTC video stream conforming to the WebRTC protocol;

[0075] The stream split screen display module 240 is arranged to split screen and display the WebRTC video stream in a split screen interface generated in a preset split screen format and based on the Video tag of the local browser HTML5.

[0076] Figure 3 is a hardware structure schematic diagram of the stream pulling device of the embodiment of the application. As shown in Figure 3 As shown, the screen projection server comprises a memory 310 and a processor 320, wherein the memory 310 and the processor 320 are in communication; for example, the memory 310 and the processor 320 are in communication through a communication bus 330, the memory 310 is configured to store a computer program, and the processor 320 executes the computer program to implement the method shown in the embodiment.

[0077] Optionally, the stream-pulling device can further comprise a transmitter and / or a receiver.

[0078] Optionally, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, a PLC (programmable logic controller), an FPGA (field-programmable gate array), a DSP (digital signal processor), or an ASIC (application specific integrated circuit). The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the present application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0079] The embodiment of the present application provides a video stream split-screen display system, as shown in the figure. Figure 4 As shown, the video stream split-screen display system 4 comprises a stream-pushing client 410, a stream media server 420, and a stream-pulling device 430 as described above.

[0080] The embodiment of the present application provides a storage medium, which is configured to store a computer program, and the computer program is configured to implement the video stream split-screen display method described in any method embodiment.

[0081] The chip is used for supporting a receiving device (for example, a terminal device, a network device, etc.) to implement the functions shown in the embodiments of the present application, and is specifically used in a chip system. The chip system can be composed of the chip, or can include the chip and other discrete devices. When the chip is used in the receiving device to implement the above method, the chip includes a processing unit, and further, the chip can also include a communication unit. The processing unit can be, for example, a processor, and when the chip includes the communication unit, the communication unit can be, for example, an input / output interface, a pin, or a circuit, etc. The processing unit performs all or part of the actions performed by each processing module in the embodiments of the present application, and the communication unit can perform corresponding receiving or sending actions. In another specific embodiment, the processing module of the receiving device in the embodiments of the present application can be the processing unit of the chip, and the receiving module or the sending module of the control device is the communication unit of the chip.

[0082] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device (apparatus) or computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing computer usable program code.

[0083] The present application is described with reference to flowcharts according to the methods, devices (apparatuses) and computer program products of the embodiments of the present application. It should be understood that each flow in the flowcharts can be implemented by computer program instructions.

[0084] These computer program instructions can be stored in a computer readable memory capable of directing a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices, which implement the functions specified in the flow Figure 1 .

[0085] These computer program instructions can also be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in the flow Figure 1 .

[0086] Another embodiment of the present application relates to a non-volatile storage medium for storing a computer readable program for a computer to execute part or all of the method embodiments described above.

[0087] That is, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by a program stored in a storage medium, including a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0088] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.< / port> < / ip>

Claims

1. A method for split-screen display of a video stream, applied to a streaming device, wherein the streaming device is equipped with a browser that supports the WebRTC protocol for real-time web communication, characterized in that, The method comprises: Preforming a local pull stream configuration to generate a stream list, wherein the stream list comprises one or more stream list items, and each stream list item comprises a pull stream address of a corresponding video stream in a stream media server; In response to a trigger instruction for playing a video stream corresponding to a stream list item in the stream list, obtaining the pull stream address of the video stream in the stream media server, and pulling the video stream from the stream media server based on the pull stream address of the video stream in the stream media server, wherein the video stream is an original RTSP video stream sent to the stream media server by a push stream client, or an RTSP video stream after transcoding by the stream media server; Decoding and unpacking the video stream to generate a WebRTC video stream conforming to a WebRTC protocol; Displaying the WebRTC video stream in a split screen interface generated according to a preset split screen format and based on a local browser HTML5 Video tag; The preformed local pull stream configuration to generate a stream list specifically comprises: Obtaining stream configuration data for each video stream input by a user to generate a corresponding stream list item; Generating the stream list based on the stream list items of each video stream; The stream configuration data comprises a video stream name, a pull stream address, and a stream play type, and the stream play type comprises continuous play and click play, wherein the stream play type is used to determine the timing of the stream media server obtaining the video stream, the continuous play means that the stream media server has continuously obtained the video stream from the push stream client before the pull stream device plays the video stream, and the click play means that the stream media server starts to obtain the video stream from the push stream client only when the pull stream device plays the video stream; Each display window comprises an area for responding to the addition and playing of a video stream, and is used to receive a trigger instruction for adding and playing a video stream corresponding to a stream list item in the stream list by a user in the display window, in an interface provided with menu items or list items of a split screen display interface, after clicking a menu item or a list item of the split screen display interface, the split screen interface is displayed, and after entering the split screen interface, no video is played by default, after clicking the area for adding a video stream in one of the display windows and selecting one of the stream list items in the pop-up stream list, the video stream corresponding to the stream list item is added and played in the corresponding display window.

2. The method of claim 1, wherein, The pulling of the video stream from the stream media server based on the pull stream address of the video stream corresponding to the stream list item in the stream media server specifically comprises: Establishing a WebRTC connection channel between a WebRTC protocol interface in a local browser and the stream media server, and pulling the video stream through the WebRTC connection channel.

3. The method of claim 1, wherein, The split screen display of the WebRTC video stream in the split screen interface generated according to the preset split screen format and based on the local browser HTML5 Video tag specifically comprises: According to a preset split-screen format, the split-screen interface is divided into a plurality of independent display windows based on a local browser HTML5 Video tag, each display window corresponding to a Video and used for playing a WebRTC video stream.

4. A video stream split-screen display device, arranged in a streaming device, wherein the streaming device is installed with a browser supporting Web Real-Time Communication (WebRTC) protocol, characterized in that, The device comprises: a stream list generation module configured to generate a stream list in advance by locally performing a pull stream configuration, wherein the stream list comprises one or more stream list items, and each stream list item comprises a pull stream address of a corresponding video stream in a stream media server; a pull stream module configured to, in response to a trigger instruction for playing a video stream corresponding to a stream list item in the stream list, acquire the pull stream address of the video stream corresponding to the stream list item in the stream media server, and pull the video stream from the stream media server based on the pull stream address of the video stream corresponding to the stream list item in the stream media server, wherein the video stream is an original RTSP video stream sent by a push stream client to the stream media server or an RTSP video stream transcoded by the stream media server; a stream decoding module configured to decode and unpack the video stream to generate a WebRTC video stream conforming to a WebRTC protocol; a stream split-screen display module configured to split-screen display the WebRTC video stream in a split-screen interface generated according to a preset split-screen format and based on a local browser HTML5 Video tag; The stream list generation module is further configured to: acquire stream configuration data for each video stream input by a user to generate a corresponding stream list item; generate the stream list based on the stream list items of each video stream; wherein the stream configuration data comprises a video stream name, a pull stream address and a stream playing type, and the stream playing type comprises continuous playing and click playing, the stream playing type being used to determine a timing at which the stream media server acquires a video stream, the continuous playing being that the stream media server continuously acquires the video stream from the push stream client before a pull stream device plays the video stream, and the click playing being that the stream media server starts to acquire the video stream from the push stream client only when the pull stream device plays the video stream; wherein each display window comprises an area for responding to adding and playing a video stream, used to receive a trigger instruction for adding and playing a video stream corresponding to a stream list item in the stream list by a user in the display window, in an interface provided with menu items or list items of a split-screen display interface, after a menu item or a list item of the split-screen display interface is clicked, the split-screen interface is displayed, and after the split-screen interface is entered, no video is played by default, after an area for adding a video stream in one of the display windows is clicked and one of the stream list items in a pop-up stream list is selected, the video stream corresponding to the stream list item is added and played in the corresponding display window.

5. A stream drawing device, characterized by A device comprises a memory and a processor, the processor executes program instructions in the memory, and is used to implement the method of any one of claims 1-3.

6. A video stream split screen display system, characterized by, The device comprises: a push stream client, a stream media server and a pull stream device as claimed in claim 5.

7. A storage medium, characterized by The storage medium is configured to store a computer program, and the computer program is configured to implement the method in any one of claims 1-3 when executed by a processor.

Images