Multi-device plug-flow live broadcast method and related device

Abstract

The invention provides a multi-device plug-flow live broadcast method and a related device, relates to the technical field of software, and is applied to a main plug-flow end, responds to an auxiliary live broadcast operation, and establishes a local area network transmission channel with an auxiliary plug-flow end. Receiving first coded data of the secondary stream pushing end based on the local area network transmission channel, and decoding the first coded data into first original data; meanwhile, acquiring locally acquired second original data; and synthesizing the first original data and the second original data to obtain synthesized data, encoding the synthesized data into second encoded data, and uploading the second encoded data to the server side, so that the server side decodes the second encoded data into the synthesized data and issues the synthesized data to the audience side. According to the method, the local area network transmission channel between the main stream pushing end and the auxiliary stream pushing end is established, data synthesis can be completed by the main stream pushing end, then the data are transmitted to the server end, stream pushing is performed to the audience end, the main stream pushing end bears part of tasks, and the hardware resource pressure of the server end is reduced.

Description

Technical Field

[0001] The present application relates to the field of software technology, and in particular to a method for live streaming of multiple devices and related devices. Background Art

[0002] At present, multi-device live broadcast has gradually become the mainstream of live broadcast. The main push stream sends the host's audio and picture to the server, and the secondary push stream also sends the host's PPT, video, etc. to the server. The server synthesizes the data from the main and secondary push streams and then sends them to the audience.

[0003] However, since the server needs to undertake high-computation tasks such as audio and video synthesis, transcoding, and mixing, it places high demands on hardware resources. Summary of the invention

[0004] In view of the above problems, the present application provides a multi-device live streaming method and related apparatus to achieve the purpose of data synthesis from the main streaming end. The specific scheme is as follows:

[0005] A first aspect of the present application provides a method for live streaming of multiple devices, wherein the method is applied to a main streaming end, and the method comprises:

[0006] Respond to auxiliary live broadcast operations and establish a LAN transmission channel with the secondary streaming end;

[0007] Receiving the first coded data of the secondary push stream end based on the local area network transmission channel, and decoding the first coded data into first original data; at the same time, obtaining the second original data collected locally;

[0008] The first original data and the second original data are synthesized to obtain synthesized data, and the synthesized data is encoded into second encoded data and uploaded to the server, so as to achieve: the server decodes the second encoded data into the synthesized data and sends it to the viewer.

[0009] In a possible implementation, establishing a local area network transmission channel with the secondary push stream end includes:

[0010] Acquire the IP address of the local device, and generate a streaming address according to the IP address, so that the secondary streaming end is configured with the streaming address;

[0011] A connection request is received from the secondary streaming end based on the configuration result of the streaming address, the configuration result of the streaming address is verified, and after the verification is passed, the local area network transmission channel is established.

[0012] In a possible implementation, synthesizing the first original data and the second original data to obtain synthesized data includes:

[0013] caching the first original data and the second original data;

[0014] The first original data and the second original data of the preset time length that are cached are synthesized to obtain the synthesized data.

[0015] In a possible implementation, synthesizing the first original data and the second original data to obtain synthesized data includes:

[0016] Obtaining the transmission delay of the local area network transmission channel;

[0017] intercepting third original data aligned with the second original data from the first original data according to the transmission delay;

[0018] The second original data and the third original data are synthesized to obtain the synthesized data.

[0019] In a possible implementation, the second original data includes first audio data, the third original data includes second audio data, and the synthesizing the second original data and the third original data to obtain the synthesized data includes:

[0020] synthesizing the first audio data and the second audio data to obtain synthesized audio data;

[0021] Overlapping audio data in the synthesized audio data is eliminated.

[0022] In a possible implementation, the second original data includes first picture data, the third original data includes second picture data, and the synthesizing the second original data and the third original data to obtain the synthesized data includes:

[0023] synthesizing the first picture data and the second picture data to obtain synthesized picture data, wherein the first picture data is located in a first display area and the second picture data is located in a second display area;

[0024] In response to the screen preview operation, the first display area and / or the second display area are adjusted.

[0025] A second aspect of the present application provides a multi-device live streaming device, wherein the multi-device live streaming device is applied to a main streaming end, and the multi-device live streaming device comprises:

[0026] The channel establishment module is used to respond to the auxiliary live broadcast operation and establish a local area network transmission channel with the secondary push stream end;

[0027] The data synthesis module is used to receive the first encoded data of the secondary push stream end based on the local area network transmission channel, and decode the first encoded data into the first original data; at the same time, obtain the second original data collected locally; synthesize the first original data and the second original data to obtain synthesized data, and encode the synthesized data into the second encoded data and upload it to the server, so as to achieve: the server decodes the second encoded data into the synthesized data and sends it to the audience end.

[0028] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the multi-device live streaming method of the first aspect or any implementation of the first aspect.

[0029] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0030] The memory is used to store computer programs;

[0031] The processor is used to execute the computer program so that the electronic device can implement the multi-device live streaming method according to the first aspect or any implementation method of the first aspect.

[0032] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the multi-device live streaming method of the first aspect or any implementation method of the first aspect.

[0033] By means of the above technical scheme, the present application provides a multi-device live streaming method and related apparatus, which are applied to the main streaming end, respond to the auxiliary live streaming operation, and establish a local area network transmission channel with the auxiliary streaming end; based on the local area network transmission channel, the first encoded data of the auxiliary streaming end is received, and the first encoded data is decoded into the first original data; at the same time, the second original data collected locally is obtained; the first original data and the second original data are synthesized to obtain synthesized data, and the synthesized data is encoded into the second encoded data and then uploaded to the server end, so as to achieve: the server end decodes the second encoded data into the synthesized data and then sends it to the audience end. In the present application, by establishing a local area network transmission channel between the main and auxiliary streaming ends, the main streaming end can complete the data synthesis and then transmit it to the server end, so as to push the stream to the audience end, which makes the main streaming end undertake part of the task and reduces the hardware resource pressure of the server end. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0035] Figure 1 A flowchart of a method for live streaming of multiple devices provided in an embodiment of the present application;

[0036] Figure 2 A partial flow chart of a multi-device live streaming method provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of another part of the flow chart of a multi-device live streaming method provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of another part of the flow chart of a multi-device live streaming method provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of the structure of a multi-device live streaming device provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0042] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0043] The terms "first", "second" etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0044] For ease of understanding, the following first describes the relevant concepts involved in this application:

[0045] Main streaming end: The device used by the host to broadcast, which collects the host's audio and image through the camera and microphone respectively.

[0046] Secondary streaming end: Another device used by the host to display PPT, play videos and other auxiliary live broadcasts.

[0047] Viewer end: the device used to watch the host’s live broadcast.

[0048] Streaming SDK: A development kit that encapsulates the live streaming function. Other developers can call the streaming SDK to implement the live streaming function.

[0049] Live broadcast is divided into single-device live broadcast and multi-device live broadcast. Single-device live broadcast adopts RTMP protocol, and multi-device live broadcast adopts RTC protocol.

[0050] RTMP protocol is a TCP-based streaming protocol developed by Adobe. Data is transmitted in blocks, with a certain buffer during transmission to ensure smoothness but introduce delays. Its advantages include good stability (based on TCP protocol, good transmission reliability), strong compatibility (support for multiple media formats and older streaming servers), and mature ecology (widely used in live broadcast scenarios, with rich tools and documents). Its disadvantages include high latency (usually 1-3 seconds, not suitable for high real-time scenarios).

[0051] The RTC protocol (represented by WebRTC) is a real-time communication protocol mainly based on UDP. Its advantages include low latency (generally within 100 milliseconds), and its disadvantages include high complexity (need to solve the problems of NAT penetration and P2P connection, and high deployment threshold), stability dependence on the network (UDP transmission may cause fluctuations in audio and video quality in a high packet loss environment), and hardware dependence (certain requirements for terminal equipment and network performance).

[0052] The RTC protocol can support the process of server-side synthesis (such as the synthesis processing of multiple audio and video streams) and then sending it down. This mode is usually called the MCU (multi-point control unit) mode. When multiple streaming ends push streams, each uploads its own audio and picture to the server; the server synthesizes the multiple audio and video streams after receiving them, including mixing and video splicing; then the audience end pulls the stream to the server end.

[0053] The existing solution is to use the MCU mode of the RTC protocol for multi-device live broadcasting. In simple terms, the main push stream sends the host's audio and video to the server, and the secondary push stream sends the host's PPT, video, etc. to the server. The server synthesizes the data from the main and secondary push streams and then sends it to the audience. However, since the server needs to undertake high-computation tasks such as audio and video synthesis, transcoding, and mixing, this places high demands on hardware resources.

[0054] In order to solve the above problems, the embodiment of the present application provides a method for live streaming of multiple devices. The method for live streaming of multiple devices in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0055] See also Figure 1 , Figure 1 A flowchart of a method for live streaming of multiple devices provided in an embodiment of the present application is shown below. Figure 1 As shown, a multi-device live streaming method provided in an embodiment of the present application is applied to a main streaming end and integrated in a streaming SDK of the main streaming end, and may include steps S101 to S103, and these steps are described in detail below.

[0056] S101, responding to the auxiliary live broadcast operation, and establishing a local area network transmission channel with the secondary push stream end.

[0057] In the embodiment of the present application, the main stream pusher establishes a live streaming channel with the audience through the server. Specifically, the RTMP protocol for single-device live streaming can be adopted. The main stream pusher sends the host's audio and picture to the server, and the audience pulls data from the server for display.

[0058] During the live broadcast of the main push stream, if the anchor needs to use another device for auxiliary live broadcast (such as displaying videos, displaying PPT, writing and drawing, etc.), the newly opened device can be used as the secondary push stream. In this regard, the anchor can perform auxiliary live broadcast operations on the main push stream, and the main push stream responds to the auxiliary live broadcast operation, determines the secondary push stream, and establishes a LAN transmission channel with the secondary push stream.

[0059] In a possible implementation, the LAN transmission channel between the primary and secondary streaming ends can be realized by configuring the streaming address on the secondary inference end. Figure 2 , Figure 2 A partial flow chart of a multi-device live streaming method provided in an embodiment of the present application. Figure 2 As shown, an embodiment of the present application provides a multi-device live streaming method, wherein "establishing a local area network transmission channel with a secondary streaming end" in step S101 may include steps S1011 to S1012, and these steps are described in detail below.

[0060] S1011, obtaining the IP address of the local device, and generating a streaming address according to the IP address, so that the secondary streaming end is configured with the streaming address.

[0061] In an embodiment of the present application, the main streaming end responds to the auxiliary live broadcast operation and can start a local server, generate a corresponding streaming address according to the IP address of the local device, and use the RTMP protocol to illustrate that the streaming address can be composed of an IP address, a port address, and a device identifier of the main streaming end. For example, the streaming address is "RTMP: / / AAA:BBB / CCC", where "AAA" is the local IP address of the main streaming end, "BBB" is the port address, and "CCC" is the device identifier of the main streaming end.

[0062] After generating the streaming address, the main streaming end can display the streaming address to the anchor, and the anchor configures the streaming address to the secondary streaming end. Specifically, the configuration can be done by scanning or direct input, which is not limited in this embodiment of the present application.

[0063] S1012, receiving a connection request sent by the secondary streaming end based on the configuration result of the streaming address, verifying the configuration result of the streaming address, and establishing a local area network transmission channel after the verification passes.

[0064] In the embodiment of the present application, after the anchor configures the secondary push stream end, the secondary push stream end can send a connection request to the primary push stream end based on the configuration result of the push stream address. After the primary push stream end receives the connection request from the secondary push stream end, it can verify the configuration result of the push stream address, specifically, it can verify whether the IP address, port address and device identification of the primary push stream end are consistent with the relevant information in the generated push stream address. If they are consistent, it means that the verification is passed, and the secondary push stream end is used as a local client to establish a local area network transmission channel with the secondary push stream end.

[0065] S102, receiving first coded data from the secondary push stream end based on the local area network transmission channel, and decoding the first coded data into first original data; at the same time, obtaining second original data collected locally.

[0066] In the embodiment of the present application, when the anchor starts video playback, or starts PPT, or opens the drawing function on the secondary push stream end, the secondary push stream end can obtain its original data (i.e., the first original data) by recording the screen. Then, the secondary push stream end encodes the first original data into corresponding encoded data (i.e., the first encoded data), and transmits the first encoded data to the main push stream end based on the local area network transmission channel.

[0067] The main push stream end receives the first coded data from the auxiliary push stream end based on the local area network transmission channel, and decodes the first coded data into the first original data. At the same time, the main push stream end can also collect the original data played locally (i.e., the second original data), and the second original data is the audio picture of the main push stream end.

[0068] It should be noted that the secondary push stream end compresses the first original data (the audio and picture of the secondary push stream end) into the first encoded data through encoding, and the secondary push stream end renders and restores the first encoded data into the first original data through decoding. The encoding method of the secondary push stream end and the decoding method of the primary push stream end are pre-configured.

[0069] S103, synthesizing the first original data and the second original data to obtain synthesized data, encoding the synthesized data into second encoded data and uploading it to the server, so as to achieve: the server decodes the second encoded data into the synthesized data and sends it to the viewer.

[0070] In an embodiment of the present application, after obtaining the first original data and the second original data, the main push stream end can synthesize the two to obtain corresponding synthesized data, which may specifically include audio synthesis and picture synthesis. Furthermore, the main push stream end encodes the synthesized data (i.e., the second encoded data) and transmits it to the server end based on the push stream live broadcast channel established with the server end. The server end renders and restores the second encoded data into synthesized data by decoding, and sends the synthesized data to the audience end, thereby enabling the audience end to pull the synthesized data, and the audience end can see the synthesized audio and picture without making any changes. The audience can see the audio picture of the main push stream end and the audio picture of the secondary push stream end at the same time. The encoding method of the main push stream end and the decoding method of the server end are pre-configured.

[0071] It should be noted that in the embodiment of the present application, encoding operations are required before the secondary push stream end transmits to the main push stream end, and before the main push stream end transmits to the server end, which can compress the data to reduce the data volume and make it more suitable for network transmission.

[0072] In a possible implementation, the original data of the primary and secondary streaming ends within a certain period of time can be synthesized through data caching. Figure 3 , Figure 3 Another partial flow chart of a multi-device live streaming method provided in an embodiment of the present application. Figure 3 As shown, an embodiment of the present application provides a multi-device live streaming method, wherein "synthesizing the first original data and the second original data to obtain synthesized data" in step S103 may include steps S1031 to S1032, and these steps are described in detail below.

[0073] S1031, cache the first original data and the second original data.

[0074] In the embodiment of the present application, after obtaining the first original data and the second original data, the two can be cached separately and cached to different addresses.

[0075] S1032, synthesize the cached first original data of the preset time length and the second original data to obtain synthesized data.

[0076] In an embodiment of the present application, taking the preset duration of 100 ms as an example, after 100 ms of first original data and 100 ms of second original data are cached, the 100 ms of first original data and the 100 ms of second original data can be synthesized to obtain 100 ms of synthesized data.

[0077] In a possible implementation, since there is a delay in the local area network transmission, the first original data and the second original data need to be aligned before the data is synthesized. Figure 4 , Figure 4 Another partial flow chart of a multi-device live streaming method provided in an embodiment of the present application. Figure 4 As shown, an embodiment of the present application provides a multi-device live streaming method, wherein "synthesizing the first original data and the second original data to obtain synthesized data" in step S103 may include steps S1033 to S1035, and these steps are described in detail below.

[0078] S1033, obtaining the transmission delay of the local area network transmission channel.

[0079] In an embodiment of the present application, after the LAN transmission channel is established, the secondary push stream end can send test data to the primary push stream end once. The test data can include a timestamp of the sending time of the secondary push stream end. When the primary push stream end receives the test data, it can record the local timestamp of the receiving time, and then calculate the transmission delay of the LAN transmission channel based on the local timestamp and the timestamp in the test data.

[0080] Of course, in actual applications, the transmission delay of the LAN transmission channel can also be determined while establishing the LAN transmission channel. Specifically, when the secondary push stream end sends a connection request to the primary push stream end, the request may include a timestamp of the time when the secondary push stream end sends the request, and when the primary push stream end receives the connection request, it may record the local timestamp of the receiving time, and then calculate the transmission delay of the LAN transmission channel based on the local timestamp and the timestamp in the test data.

[0081] S1034: extract third original data aligned with the second original data from the first original data according to the transmission delay.

[0082] In an embodiment of the present application, after the auxiliary live broadcast is turned on, there is a transmission delay when the first original data reaches the main push stream end via the local area network transmission channel. Therefore, the first original data when arriving at the main push stream end should be sent by the auxiliary push stream end before the transmission delay. Therefore, it is necessary to intercept the data after the transmission delay in the first original data, and align the intercepted third original data with the second original data.

[0083] S1035, synthesize the second original data and the third original data to obtain synthesized data.

[0084] In the embodiment of the present application, the main streaming end synthesizes the second original data with the third original time, performs audio synthesis at the same time, and performs picture synthesis at the same time, so as to output the synthesized audio and picture at different times.

[0085] In the scenario where a host uses two devices to broadcast live, this live broadcast method has the problem of audio overlap. The audio played by the secondary host will not only be transmitted to the main push stream end, but also collected by the microphone of the main push stream end. When the audio is synthesized, the two audio channels have the same sound, which will cause overlap, resulting in poor listening effect for the audience. This requires the use of an overlapping algorithm to remove the repeated parts in the audio. In this regard, the present application uses a de-overlapping algorithm after aligning the first original data with the second original data.

[0086] The second original data includes the first audio data, and the third original data includes the second audio data. The embodiment of the present application provides a multi-device live streaming method, wherein step S1035 "synthesizes the second original data and the third original data to obtain synthesized data" can be performed using the following steps:

[0087] The first audio data and the second audio data are synthesized to obtain synthesized audio data; and overlapping audio data in the synthesized audio data are eliminated.

[0088] In the embodiment of the present application, after the first audio data and the second audio data are aligned, the two are synthesized to obtain synthesized audio data, and a de-overlapping algorithm is used to eliminate the overlapping audio data (ie, overlapping audio data) in the synthesized audio data.

[0089] In a possible implementation, a screen preview function can also be provided for the anchor. The second original data includes the first screen data, and the third original data includes the second screen data. In a multi-device live streaming method provided in an embodiment of the present application, step S1035 "synthesizes the second original data and the third original data to obtain synthesized data" can be implemented by the following steps:

[0090] The first picture data and the second picture data are synthesized to obtain synthesized picture data, in which the first picture data is located in a first display area and the second picture data is located in a second display area; in response to the picture preview operation, the first display area and / or the second display area are adjusted.

[0091] In the embodiment of the present application, after aligning the first screen data and the second screen data, the two are synthesized to obtain synthesized screen data, in which the first screen data and the second screen data have their own display areas, that is, the first screen data is located in the first display area, and the second screen data is located in the second display area, and the first display area and the second display area can be overlapped. Generally speaking, the screen of the secondary push stream end can be used as a background, and the screen of the main push stream end can be located at a certain position. Therefore, in the initial state, the second display area can be the interface area of ​​the main push stream end, and the first display area is located at a certain position of the second display area, and the area of ​​the second display area is larger than that of the first display area.

[0092] The host can also adjust the position and size of the first display area and the second display area in the composite picture data by dragging on the main push stream end. In this regard, the embodiment of the present application provides a picture preview function, and the main push stream end renders and displays the composite picture data, and in response to the host's picture preview operation, adjusts the position / size of any one or more of the first display area and the second display area.

[0093] Through the above description, the embodiment of the present application provides a multi-device live streaming method, which can complete data synthesis by the main streaming end and then transmit it to the server end, so as to stream to the audience end, which makes the main streaming end take on part of the tasks and reduces the hardware resource pressure of the server end. The present application has the following advantages:

[0094] 1) The LAN mixing function is realized with only a few changes on the basis of the traditional streaming SDK; 2) The LAN mixing makes similar teaching live broadcasts clearer and more flexible; 3) The cost of the server is greatly reduced.

[0095] A method for live streaming of multiple devices provided in an embodiment of the present application is described above. A device for executing the method for live streaming of multiple devices is described below.

[0096] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a multi-device live streaming device provided in an embodiment of the present application. Figure 5 As shown, a multi-device live streaming device provided in an embodiment of the present application includes:

[0097] The channel establishment module 501 is used to respond to the auxiliary live broadcast operation and establish a local area network transmission channel with the secondary push stream end;

[0098] The data synthesis module 502 is used to receive the first encoded data from the secondary push stream end based on the local area network transmission channel, and decode the first encoded data into the first original data; at the same time, obtain the second original data collected locally; synthesize the first original data and the second original data to obtain synthesized data, and encode the synthesized data into the second encoded data and upload it to the server end, so as to achieve: the server end decodes the second encoded data into the synthesized data and sends it to the audience end.

[0099] In a possible implementation, the channel establishing module 501 for establishing a local area network transmission channel with the secondary push stream end is specifically used for:

[0100] Obtain the IP address of the local device and generate a streaming address based on the IP address so that the secondary streaming end can configure the streaming address; receive a connection request sent by the secondary streaming end based on the configuration result of the streaming address, verify the configuration result of the streaming address, and establish a LAN transmission channel after the verification is passed.

[0101] In a possible implementation, the data synthesis module 502 for synthesizing the first original data and the second original data to obtain synthesized data is specifically configured to:

[0102] The first original data and the second original data are cached; and the cached first original data and the second original data of a preset time length are synthesized to obtain synthesized data.

[0103] In a possible implementation, the data synthesis module 502 for synthesizing the first original data and the second original data to obtain synthesized data is specifically configured to:

[0104] Acquire the transmission delay of the local area network transmission channel; intercept the third original data aligned with the second original data from the first original data according to the transmission delay; synthesize the second original data and the third original data to obtain synthesized data.

[0105] In a possible implementation, the second original data includes the first audio data, the third original data includes the second audio data, and the data synthesis module 502 for synthesizing the first original data and the second original data to obtain synthesized data is specifically configured to:

[0106] The first audio data and the second audio data are synthesized to obtain synthesized audio data; and overlapping audio data in the synthesized audio data are eliminated.

[0107] In a possible implementation, the second original data includes the first picture data, the third original data includes the second picture data, and the data synthesis module 502 for synthesizing the first original data and the second original data to obtain synthesized data is specifically used to:

[0108] The first picture data and the second picture data are synthesized to obtain synthesized picture data, in which the first picture data is located in a first display area and the second picture data is located in a second display area; in response to the picture preview operation, the first display area and / or the second display area are adjusted.

[0109] It should be noted that the detailed functions of each module in the embodiment of the present application can be found in the corresponding public part of the above-mentioned multi-device live streaming method embodiment, which will not be repeated here.

[0110] The present application also provides an electronic device in an embodiment. Figure 6 , Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0111] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0112] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0113] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any multi-device live streaming method provided in the embodiment of the present application.

[0114] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any multi-device live streaming method provided in an embodiment of the present application.

[0115] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0116] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0117] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0118] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for live streaming of multiple devices, characterized in that: The multi-device live streaming method is applied to the main streaming end, and the multi-device live streaming method includes: Respond to auxiliary live broadcast operations and establish a LAN transmission channel with the secondary streaming end; Receiving the first coded data of the secondary push stream end based on the local area network transmission channel, and decoding the first coded data into first original data; at the same time, obtaining the second original data collected locally; The first original data and the second original data are synthesized to obtain synthesized data, and the synthesized data is encoded into second encoded data and uploaded to the server, so as to achieve: the server decodes the second encoded data into the synthesized data and sends it to the viewer.

2. The method for live streaming of multiple devices according to claim 1, characterized in that: The step of establishing a local area network transmission channel with the secondary push stream end includes: Acquire the IP address of the local device, and generate a streaming address according to the IP address, so that the secondary streaming end is configured with the streaming address; A connection request is received from the secondary streaming end based on the configuration result of the streaming address, the configuration result of the streaming address is verified, and after the verification is passed, the local area network transmission channel is established.

3. The method for live streaming of multiple devices according to claim 1, characterized in that: The synthesizing the first original data and the second original data to obtain synthesized data includes: caching the first original data and the second original data; The first original data and the second original data of the preset time length that are cached are synthesized to obtain the synthesized data.

4. The method for live streaming of multiple devices according to claim 1, characterized in that: The synthesizing the first original data and the second original data to obtain synthesized data includes: Obtaining the transmission delay of the local area network transmission channel; intercepting third original data aligned with the second original data from the first original data according to the transmission delay; The second original data and the third original data are synthesized to obtain the synthesized data.

5. The method for live streaming of multiple devices according to claim 4, characterized in that: The second original data includes first audio data, the third original data includes second audio data, and the synthesizing the second original data and the third original data to obtain the synthesized data includes: synthesizing the first audio data and the second audio data to obtain synthesized audio data; Overlapping audio data in the synthesized audio data is eliminated.

6. The method for live streaming of multiple devices according to claim 4, characterized in that: The second original data includes first picture data, the third original data includes second picture data, and the synthesizing the second original data and the third original data to obtain the synthesized data includes: synthesizing the first picture data and the second picture data to obtain synthesized picture data, wherein the first picture data is located in a first display area and the second picture data is located in a second display area; In response to the screen preview operation, the first display area and / or the second display area are adjusted.

7. A multi-device live streaming device, characterized in that: The multi-device live streaming device is applied to the main streaming end, and the multi-device live streaming device includes: The channel establishment module is used to respond to the auxiliary live broadcast operation and establish a local area network transmission channel with the secondary push stream end; The data synthesis module is used to receive the first encoded data of the secondary push stream end based on the local area network transmission channel, and decode the first encoded data into the first original data; at the same time, obtain the second original data collected locally; synthesize the first original data and the second original data to obtain synthesized data, and encode the synthesized data into the second encoded data and upload it to the server, so as to achieve: the server decodes the second encoded data into the synthesized data and sends it to the audience end.

8. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the multi-device live streaming method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the multi-device live streaming method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the multi-device live streaming method as described in any one of claims 1 to 6.

Images