Live connection and confluence switching method and system

Abstract

The present application discloses a method for switching live co-hosting video mixing, which includes: before the co-hosting starts, obtaining the hardware device evaluation parameters of the host client according to the hardware influencing factors; after the co-hosting starts, obtaining the real-time network evaluation parameters during the co-hosting according to the real-time influencing factors; combining the hardware device evaluation parameters and the real-time network evaluation parameters to obtain a real-time evaluation result, determining the currently applicable video mixing method according to the real-time evaluation result and a preset threshold; and performing audio and video mixing for the live co-hosting according to the determined video mixing method. The present application also discloses a live co-hosting video mixing switching system, an electronic device, and a computer-readable storage medium. Thereby, it is possible to dynamically switch different video mixing methods to adapt to the current live co-hosting scenario, and improve the host co-hosting experience and the viewer viewing experience.

Description

Technical Field

[0001] The present application relates to the field of live broadcast technology, and in particular, to a live connection and stream merging switching method, system, electronic device, and computer-readable storage medium. Background Art

[0002] With the development of Internet technology, video live broadcast has become a trend, and more and more live broadcast scenarios have emerged. Among them, live connection is a frequently occurring live broadcast scenario. The connection refers to multiple people having real-time audio and video communication through the network, and merging multiple live broadcast images and audio and then pushing them to the viewers in the live broadcast room for viewing.

[0003] For ordinary live broadcasts, the images pushed to the live broadcast room are generated by scene rendering that allows the anchor to freely edit and are directly provided by the anchor client. When making a connection, it is necessary to splice the live broadcast images cropped by the local anchor client with the live broadcast images cropped by the other client to form a merged connection image and push it to the live broadcast room.

[0004] Currently, there are mainly two types of current live connection and stream merging methods. One is to perform merging on the connection server side, and the other is to perform merging on the local device of the anchor side. Both of these methods have their respective applicable application scenarios, advantages, and disadvantages. However, at present, there is a lack of a reliable switching method in the industry, and it is impossible to dynamically select an appropriate merging method according to the specific situation of the anchor side, resulting in a poor user experience. Summary of the Invention

[0005] The main purpose of the present application is to propose a live connection and stream merging switching method, system, electronic device, and computer-readable storage medium, aiming to solve the problem of how to provide a reliable connection and stream merging method switching scheme to dynamically select an appropriate merging method, thereby improving the merging speed and effect.

[0006] To achieve the above object, an embodiment of the present application provides a live connection and stream merging switching method, which is applied to an anchor client. The method includes:

[0007] After the connection starts, obtain real-time network evaluation parameters during the connection process according to preset real-time influencing factors;

[0008] Obtain a real-time evaluation result according to the real-time network evaluation parameters, and determine the currently applicable merging method according to the real-time evaluation result and a preset threshold;

[0009] Perform audio and video merging of the live connection according to the determined merging method.

[0010] Optionally, the method further includes:

[0011] Before the co-hosting starts, obtain the hardware device evaluation parameters of the host client according to the preset hardware influencing factors; and

[0012] Obtain the real-time evaluation result by combining the hardware device evaluation parameters and the real-time network evaluation parameters.

[0013] Optionally, the audio-visual stream merging for the live co-hosting according to the determined merging method includes:

[0014] In the case where it is determined according to the real-time evaluation result and the preset threshold that the first merging method is applicable currently, merge the audio-visual data of all parties in the current live co-hosting in the first merging method;

[0015] In the case where it is determined according to the real-time evaluation result and the preset threshold that the second merging method is applicable currently, switch from the first merging method to the second merging method, and merge the audio-visual data of all parties in the current live co-hosting in the second merging method.

[0016] Optionally, the merging methods include local merging on the host client and server merging.

[0017] Optionally, the local merging includes:

[0018] Obtain the local screen and the screen of the other party in the co-hosting;

[0019] Merge the local screen and the screen of the other party through texture rendering to obtain the co-hosting scene screen;

[0020] Obtain the local sound source and the sound source of the other party;

[0021] Process the local sound source and the sound source of the other party through different audio tracks to obtain the co-hosting audio;

[0022] Push the co-hosting scene screen and the co-hosting audio to the content delivery network.

[0023] Optionally, the server merging includes:

[0024] Obtain the local screen and the local sound source;

[0025] Send the local screen and the local sound source to the co-hosting server, so that the co-hosting server merges the local screen and the screen of the other party in the co-hosting to obtain the co-hosting scene screen, merges the local sound source and the sound source of the other party to obtain the co-hosting audio, and pushes the co-hosting scene screen and the co-hosting audio to the content delivery network.

[0026] Optionally, determining the currently applicable merging method according to the real-time evaluation result and the preset threshold includes:

[0027] In the case where the real-time evaluation result is greater than the preset threshold, it is determined that the local merging method is currently applicable;

[0028] In the case where the real-time evaluation result is less than or equal to the preset threshold, it is determined that the server merging method is currently applicable.

[0029] Optionally, the first-level weights corresponding to the dynamic adjustment of the hardware device evaluation parameters and the real-time network evaluation parameters, the first preset influencing factors include model, CPU, graphics card, and memory, and the second preset influencing factors include the main and guest network conditions, push and pull stream frame rates, decoding and rendering conditions, and audio and video freezing rates. Each influencing factor corresponds to the second-level weight of dynamic adjustment.

[0030] In addition, to achieve the above object, an embodiment of the present application further provides a live connection and merging switching system, and the system includes:

[0031] An acquisition module, configured to obtain real-time network evaluation parameters during the connection process according to preset real-time influencing factors after the connection starts;

[0032] A judgment module, configured to obtain a real-time evaluation result according to the real-time network evaluation parameters, and determine the currently applicable merging method according to the real-time evaluation result and the preset threshold;

[0033] A processing module, configured to perform audio and video merging of the live connection according to the determined merging method, and push the merged data to the content distribution network.

[0034] To achieve the above object, an embodiment of the present application further provides an electronic device, and the electronic device includes: a memory, a processor, and a live connection and merging switching program stored on the memory and executable on the processor. When the live connection and merging switching program is executed by the processor, the live connection and merging switching method as described above is implemented.

[0035] To achieve the above object, an embodiment of the present application further provides a computer-readable storage medium, and a live connection and merging switching program is stored on the computer-readable storage medium. When the live connection and merging switching program is executed by a processor, the live connection and merging switching method as described above is implemented.

[0036] The live connection and merging switching method, system, electronic device, and computer-readable storage medium proposed in the embodiments of the present application can analyze the hardware conditions and real-time network conditions of the host client during the live connection process, determine the most applicable connection and merging method based on the real-time evaluation results in each dimension, and dynamically switch different merging methods to adapt to the current live connection scenario, so as to improve the merging speed and effect, avoid problems such as freezing, black screen, and latency, and can also reduce the client pressure, ensure the connection quality, thereby enhancing the host's connection experience and the audience's viewing experience, and improving the attractiveness of the live broadcast room. Description of the Drawings

[0037] Figure 1 An application environment architecture diagram for implementing various embodiments of the present application;

[0038] Figure 2 A flowchart of a live connection and merging switching method proposed in the first embodiment of the present application;

[0039] Figure 3 When the determined merging method is local merging, Figure 2 A detailed flowchart of step S206 in

[0040] Figure 4 When the determined merging method is server merging, Figure 2 A detailed flowchart of step S206 in

[0041] Figure 5 Another form of flowchart of the live connection and merging switching method proposed in the first embodiment of the present application;

[0042] Figure 6 A hardware architecture diagram of an electronic device proposed in the second embodiment of the present application;

[0043] Figure 7 A module diagram of a live connection and merging switching system proposed in the third embodiment of the present application. Detailed Implementation Modes

[0044] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0045] It should be noted that in the embodiments of the present application, the descriptions involving "first", "second", etc. are only for descriptive purposes, and cannot be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. Additionally, the technical solutions between various embodiments may be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0046] Please refer to Figure 1 , Figure 1 which is an application environment architecture diagram for implementing various embodiments of the present application. The present application can be applied to an application environment including, but not limited to, a first client 2, a second client 4, a co-hosting server 6, an algorithm server 8, and a CDN (Content Delivery Network) 10.

[0047] Among them, the first client 2 corresponds to the client of the local user (the host), and is used to evaluate the local hardware device conditions and the real-time network conditions during the co-hosting process. This includes, before the co-hosting starts, obtaining the local hardware device evaluation parameters according to the first preset influencing factors; after the co-hosting starts, obtaining the real-time network evaluation parameters during the co-hosting process according to the second preset influencing factors; and combining the hardware device evaluation parameters and the real-time network evaluation parameters to obtain a real-time evaluation result. The first client 2 is also used to determine the currently applicable confluence method according to the real-time evaluation result and a preset threshold, and perform the audio and video confluence of the live co-hosting in the determined confluence method. The confluence methods include a local confluence method and a server confluence method. The local confluence means performing the co-hosting audio and video confluence in the first client 2, and the server confluence means performing the co-hosting audio and video confluence in the co-hosting server 6.

[0048] When the local confluence method is currently applicable, the first client 2 is also used to obtain the local screen and the other party's screen of the second client 4 in the co-hosting, confluence them to obtain the co-hosting scene screen, and obtain the local sound source and the other party's sound source of the second client 4, confluence them to obtain the co-hosting audio, and push the co-hosting scene screen and the co-hosting audio to the CDN 10.

[0049] The second client 4 corresponds to the client of the other party user (the host) participating in the co-hosting, and its functions are similar to those of the first client 2. The client can be a terminal device such as a PC (Personal Computer), a mobile phone, a tablet computer, a portable computer, a wearable device, etc.

[0050] The co-hosting server 6 is used to handle the co-hosting between the first client 2 and the second client 4, receive the live video and audio source from the first client 2 and the second client 4 respectively, and then transfer them to the second client 4 and the first client 2. When the server merging method is currently applicable, the co-hosting server 6 is further used to obtain the local video of the first client 2 and the video of the other party of the co-hosting second client 4, merge them to obtain the co-hosting scene video, and obtain the local audio source of the first client 2 and the audio source of the other party of the second client 4, merge them to obtain the co-hosting audio, and push the co-hosting scene video and the co-hosting audio to the CDN 10.

[0051] The algorithm server 8 is used to send the evaluation rules to the first client 2 and the second client 4. The evaluation rules include the influencing factors of the co-hosting quality and the corresponding weights, etc. The influencing factors include the first preset influencing factor and the second preset influencing factor. The first preset influencing factor corresponds to the local hardware device situation of the client, and the second preset influencing factor corresponds to the real-time network situation during the co-hosting process. The first client 2 and the second client 4 can evaluate the local hardware device situation according to the evaluation rules to obtain the hardware device evaluation parameters, evaluate the real-time network situation during the co-hosting process to obtain the real-time network evaluation parameters, and combine the hardware device evaluation parameters and the real-time network evaluation parameters to obtain the final real-time evaluation result.

[0052] Among them, the hardware device evaluation parameters and the real-time network evaluation parameters correspond to the first-level weights that are dynamically adjusted. The first preset influencing factor includes the model, central processing unit (CPU), graphics card, memory, etc. The second preset influencing factor includes the main and guest network situations, the push-pull stream frame rate, the decoding and rendering situation, the audio and video freezing rate, etc. Each influencing factor corresponds to the second-level weights that are dynamically adjusted. The main state refers to the local host party, and the guest state refers to the other host party of the co-hosting. For example, in the scenario where the local host A co-hosts with the other host B, in the first client 2 where the local host A is located, the local network situation is the main network situation; the network situation of the other host B is the guest network situation. If it is a scenario where the local host A co-hosts with the other hosts B and C, then both B and C are in the guest state.

[0053] The algorithm server 8 can be a separate server or merged into the co-hosting server 6 or other servers, which is not limited here. The server can be a computing device such as a rack server, a blade server, a tower server or a cabinet server, and can be an independent server or a server cluster composed of multiple servers.

[0054] Between the co-hosting server 6, the algorithm server 8, and multiple clients, such as the first client 2 and the second client 4, they are communicatively connected via a wired or wireless network for data transmission and interaction.

[0055] The CDN 10 is used to transmit the live stream push of the first client 2, the second client 4, or the co-hosting server 6 to each viewer client, that is, the client where the viewers in the live room are located, so that users of the viewer client can watch the live room. The CDN 10 is a distributed content delivery network built on a data network. It adopts streaming media server cluster technology to overcome the shortcomings of insufficient output bandwidth and concurrency ability of a single machine system, can greatly increase the number of concurrent streams supported by the system, and reduce or avoid the adverse effects brought by single point failure.

[0056] It should be noted that in addition to the case of two-party co-hosting between the first client 2 and the second client 4, there may also be a case of multi-party co-hosting in the live broadcast. Similar to the processing method of two-party co-hosting, only the audio and video sources increase from two to more. This will not be elaborated in the embodiments of the present application.

[0057] Embodiment 1

[0058] As Figure 2 shown, it is a flowchart of a live co-hosting stream merging and switching method proposed in the first embodiment of the present application. It can be understood that the flowchart in the embodiment of this method is not used to limit the order of execution steps. According to needs, some steps in this flowchart can also be added or deleted. Hereinafter, taking the host client, such as the first client, as the execution subject, this method will be described.

[0059] This method includes the following steps:

[0060] S200, before the co-hosting starts, obtain the hardware device evaluation parameters of the host client according to the preset hardware influencing factors.

[0061] In this embodiment, for the factors that may affect the co-hosting quality, they are divided into two parts in total. One is the hardware device situation of the host client, and the other is the real-time network situation during the co-hosting process. Therefore, in this embodiment, the above two parts are evaluated respectively before and after the co-hosting starts, and then the two are combined to obtain the final evaluation result.

[0062] First, the host client obtains the evaluation rules issued by the algorithm server. The evaluation rules include the influencing factors required for evaluation, the weight of each influencing factor, etc. Before the connection starts, the host client obtains data from the local hardware device according to the preset hardware influencing factors in the evaluation rules and conducts an evaluation to obtain the hardware device evaluation parameters. Among them, the hardware influencing factors are the hardware device factors that affect the connection quality, including the model, CPU, graphics card, memory, etc. Since the data of each hardware influencing factor are hardware dimension information, they can all be obtained through relevant API (Application Program Interface) interfaces.

[0063] After obtaining the data corresponding to the hardware influencing factors, classify and evaluate them respectively according to the evaluation rules, and calculate the hardware device evaluation parameters.

[0064] For example, the evaluation rule of the hardware device evaluation parameters can be:

[0065] The total score of the hardware device evaluation parameters is 100 points. If the host client is a mobile device, the model accounts for 30 points, the CPU accounts for 50 points, and the memory accounts for 20 points; if the host client is a PC device, the graphics card accounts for 30 points, the CPU accounts for 50 points, and the memory accounts for 20 points.

[0066] Among them, for the model, all devices on the market are classified according to the model platform. For example, they are divided into high, medium, and low-end models, and different categories of models correspond to different score values. For example, high-end models are 30 points, medium-end models are 20 points, and low-end models are 10 points. In addition, corresponding classifications and score values can also be set for different models of different brands and systems. For example, for the mobile phone models of the iPhone series, iPhone 14 is 30 points, iPhone 13 is 20 points, and the earlier models are 10 points, and so on.

[0067] For the CPU, it can be classified according to the model and generation of the CPU, and different categories of CPUs correspond to different score values. Taking the CPU of the iPhone as an example, A16, M1, and M2 are 50 points, A15 is 45 points, and A8 is 10 points.

[0068] For the graphics card, it can be classified according to the model and generation of the graphics card, and different categories of graphics cards correspond to different score values. Taking NVIDIA graphics cards as an example, those with a performance stronger than or equal to GTX1660 Ti are 30 points, those with a performance weaker than GTX1660 Ti but stronger than GTX 590 are 20 points, those with a performance weaker than GTX 590 but stronger than GTX560 are 10 points, and those with a performance weaker than GTX 560 are 0 points.

[0069] For memory, it can be classified according to the running memory, and different categories of memory correspond to different score values. For example, for mobile devices, if the memory is greater than or equal to 6G, it is 20 points, 5G is 15 points, 4G is 10 points, 3G is 5 points, and less than 3G is 0 points.

[0070] It should be noted that the weight corresponding to each influencing factor in the hardware influencing factors can be dynamically adjusted according to actual needs.

[0071] S202, after the co-hosting starts, obtain the real-time network evaluation parameters during the co-hosting according to the preset real-time influencing factors.

[0072] After the co-hosting starts, the host client obtains data and evaluates the real-time network situation during the co-hosting according to the evaluation rules according to the preset real-time influencing factors, and obtains the real-time network evaluation parameters. Among them, the real-time influencing factors are network factors that affect the co-hosting quality, including the network situations of the host and guest states, the push-pull stream frame rate, the decoding and rendering situation, the audio and video stuttering rate, etc. The network situations of the host and guest states can be obtained from the host client, and the push-pull stream frame rate, the decoding and rendering situation, and the audio and video stuttering rate can be obtained from the co-hosting SDK (Software Development Kit) after the co-hosting. The co-hosting SDK is the co-hosting unit in the host client (the first client), which is used to transmit the local screen and sound source to the co-hosting server, and obtain the other party's screen and sound source transmitted by the second client from the co-hosting server.

[0073] After obtaining the data corresponding to the real-time influencing factors, classification evaluation is also carried out according to the evaluation rules respectively, and the real-time network evaluation parameters are calculated.

[0074] For example, the evaluation rules for the real-time network evaluation parameters can be:

[0075] The total score of the real-time network evaluation parameters is 100 points, the network situation of the host and guest states accounts for 20 points, the push stream frame rate accounts for 20 points, the pull stream frame rate accounts for 20 points, the decoding and rendering situation accounts for 10 points, the video stuttering rate accounts for 15 points, and the audio stuttering rate accounts for 15 points.

[0076] For the network situation of the host and guest states, it can be classified according to the network status, and different categories of network status correspond to different score values. For example, the network status is divided into excellent, good, medium, and poor, and different gears correspond to different score values. For example, excellent is 20 points, good is 15 points, medium is 10 points, and poor is 0 points.

[0077] For the push-pull stream frame rate, it can be classified according to the frame rate, and different categories of frame rates correspond to different score values. The push stream frame rate refers to the number of frames pushed per second, and the pull stream frame rate refers to the number of frames pulled per second. The higher the frame rate, the smoother the picture. For example, 25 to 30 frames are 20 points, 15 to 24 frames are 15 points, 10 to 14 frames are 10 points, and less than 10 frames are 0 points.

[0078] For the decoding and rendering situation, it can be classified according to whether an anomaly occurs, where an anomaly is 0 points and normal is 10 points.

[0079] For the audio-video stuttering rate, it can be classified according to the stuttering rate, and different categories of stuttering rates correspond to different score values. For example, a stuttering rate less than 3% is 15 points, between 3% and 5% is 5 points, and greater than 5% is 0 points.

[0080] Similarly, the weight corresponding to each influencing factor in the real-time influencing factors can also be dynamically adjusted according to actual needs. And, the real-time network evaluation parameters are obtained by real-time evaluation during the co-hosting process, that is to say, they will be updated continuously during the entire co-hosting process in order to determine the currently most applicable merging method in real time in the subsequent steps.

[0081] S204, combine the hardware device evaluation parameters and the real-time network evaluation parameters to obtain a real-time evaluation result, and determine the currently applicable merging method according to the real-time evaluation result and the preset threshold.

[0082] According to the hardware device evaluation parameters obtained before the co-hosting starts and the real-time network evaluation parameters obtained by real-time evaluation during the co-hosting process, and performing combined calculation according to the evaluation rules, the final real-time evaluation result can be obtained. In this embodiment, the hardware device evaluation parameters and the real-time network evaluation parameters respectively correspond to dynamically adjustable weights. For example, the weight of the hardware device evaluation parameters is 60%, and the weight of the real-time network evaluation parameters is 40%. Multiply the hardware device evaluation parameters and the real-time network evaluation parameters by their corresponding weights and then add them together, which is the real-time evaluation result.

[0083] It should be noted that in an alternative embodiment, the weight of the real-time network evaluation parameters can also be set to 100%, and the weight of the hardware device evaluation parameters can be set to 0. That is to say, without considering the influence of the hardware device (equivalent to deleting the above step S200), only based on the real-time influencing factors, obtain the real-time network evaluation parameters during the co-hosting process, and obtain the real-time evaluation result.

[0084] Then, compare the real-time evaluation result with the preset threshold to determine the currently applicable confluence method. In this embodiment, the confluence methods include local confluence on the host client and server confluence. The local confluence means performing the co-hosting audio and video confluence locally on the host client, and the server confluence means performing the co-hosting audio and video confluence in the co-hosting server.

[0085] If the server confluence method is adopted, when switching from a normal live broadcast to a co-hosting live broadcast, the host client needs to actively cut off the stream, then establish a co-hosting session, and notify the co-hosting server to perform confluence and streaming. When switching from a co-hosting live broadcast to a normal live broadcast, the process is the opposite. During the above switching process, due to the stream cutting operation, the screen in the live broadcast room will freeze and go black, affecting the viewing experience of the users on the viewer side of the live broadcast room.

[0086] If the local confluence method is adopted, the freezing and black screen problems caused by the above switching can be avoided. However, since the confluence and streaming processes during co-hosting are both processed on the host client, it will increase the pressure on the host client and consume more client resources. If the hardware conditions of the host client are insufficient or the network condition is poor, it will also affect the co-hosting quality.

[0087] In this embodiment, the confluence methods are divided according to the score of the real-time evaluation result. For the case with a high score, the local confluence method is applicable, which can guarantee the viewing experience of the users on the viewer side; for the case with a low score, the server confluence method is applicable, which can reduce the pressure on the host client and improve the co-hosting quality. That is to say, when the real-time evaluation result is greater than the preset threshold, it is determined that the local confluence method is currently applicable; when the real-time evaluation result is less than or equal to the preset threshold, it is determined that the server confluence method is currently applicable.

[0088] For example, based on multiple experiments in actual applications, the preset threshold can be set to 74 points. When the real-time evaluation result is greater than 74 points, the local confluence method is applicable; when the real-time evaluation result is less than or equal to 74 points, the server confluence method is applicable.

[0089] S206, perform the audio and video confluence of the live co-hosting according to the determined confluence method.

[0090] After determining the currently applicable confluence method according to the real-time evaluation result and the preset threshold, the determined confluence method is used to confluence the audio and video data of the current co-hosting, and push it to the CDN, so as to be distributed to each viewer side of the live broadcast room, so that the users on the viewer side can watch the co-hosting scene of this live broadcast room.

[0091] Specifically, further refer to Figure 3, it is a schematic diagram of the refined process of step S206 when the determined confluence method is local confluence. It can be understood that this flowchart is not used to limit the order of execution steps. According to needs, some steps in this flowchart can also be added or deleted. In Figure 3 , step S206 specifically includes:

[0092] S2060, obtain the local screen and the screen of the other party in the co-hosting.

[0093] During the live broadcast, the local live broadcast screen is generated by the scene that allows the host to freely edit, and this screen will also be displayed to the host in the preview window. And the host can switch between multiple preset scenes at any time. Among them, the currently selected scene screen is the local screen.

[0094] During the co-hosting process, not only the local screen needs to be obtained, but also the screen provided by the other host (the second client) participating in the co-hosting is required, which is the screen of the other party. In this embodiment, the screen of the other party can be obtained from the co-hosting server through the co-hosting SDK.

[0095] S2062, confluence the local screen and the screen of the other party through texture rendering to obtain the co-hosting scene screen.

[0096] In this embodiment, the host client can render the local screen, the co-hosting scene screen and the streaming screen through different textures respectively. Among them, the co-hosting scene screen is formed by cropping the local screen and then combining it with the screen of the other party.

[0097] Specifically, the local screen is rendered in the first texture and can be drawn to the preview window for the local host to view. The co-hosting scene screen is rendered in the second texture and can be drawn to the co-hosting window for the local host to view. The streaming screen is rendered in the third texture, that is, the screen pushed to the live broadcast room. When the local host selects to push the screen in the preview window to the live broadcast room, the local screen in the first texture is copied to the third texture. When the local host selects to push the screen in the co-hosting window to the live broadcast room, the co-hosting scene screen in the second texture is copied to the third texture.

[0098] S2064, obtain the local sound source and the sound source of the other party.

[0099] The local sound source includes the system sound source and the current scene sound source, and the sound source of the other party can be obtained from the co-hosting server through the co-hosting SDK.

[0100] S2066, process the local sound source and the sound source of the other party through different audio tracks to obtain the co-hosting audio.

[0101] Each sound source can be selected to output audio to one or more of multiple audio tracks, and only the audio of one of the tracks will be finally output to the stream, such as the first track. Therefore, the other party's sound source can be output only to the first track, and then the audio of another track can be collected and sent to the other party's client, such as the second track.

[0102] Specifically, the system sound source and the other party's sound source are output to the first track among the multiple audio tracks. The current scene sound source is output to at least one second track and the first track among the multiple audio tracks. The audio of the first track is the co-hosting audio for pushing the stream to the live broadcast room. The audio of the second track is used to be sent to the second client through the co-hosting server.

[0103] S2068, push the co-hosting scene picture and the co-hosting audio to the CDN.

[0104] In this embodiment, the co-hosting scene picture copied from the third texture and the co-hosting audio in the first track can be encoded and then pushed to the CDN, so as to be distributed to each viewer terminal in the live broadcast room.

[0105] Further refer to Figure 4 , which is a detailed process schematic diagram of step S206 when the determined merging method is server merging. It can be understood that this flowchart is not used to limit the order of execution steps. According to needs, some steps in this flowchart can also be added or deleted. In Figure 4 In it, step S206 specifically includes:

[0106] S2061, obtain the local picture and the local sound source.

[0107] When adopting the server merging method, the host client only needs to obtain the local picture and the local sound source for the merging and pushing process, and then send them to the co-hosting server. Of course, the host client still needs to obtain the other party's picture and the other party's sound source from the co-hosting server for output to the local host to watch.

[0108] S2063, send the local picture and the local sound source to the co-hosting server, so that the co-hosting server merges the local picture and the picture of the co-hosting partner to obtain the co-hosting scene picture, merges the local sound source and the other party's sound source to obtain the co-hosting audio, and pushes the co-hosting scene picture and the co-hosting audio to the CDN.

[0109] After the host client sends the local video and the local audio source to the co-hosting server, the co-hosting server also needs to obtain the other party's video and the other party's audio source sent by the second client. Then, the co-hosting server combines the local video and the other party's video to obtain a co-hosting scene video, and combines the local audio source and the other party's audio source to obtain co-hosting audio. Finally, the co-hosting scene video and the co-hosting audio are pushed to the CDN for distribution to each viewer client in the live broadcast room. The co-hosting server can use any existing feasible technology for combination, which will not be elaborated here.

[0110] Since the real-time network evaluation parameters are evaluated and updated in real time during the co-hosting process, the real-time evaluation results will keep updating, which may cause the currently applicable combination method to change. When it is determined according to the real-time evaluation results that the currently applicable combination method has changed, it is necessary to switch from one combination method to another.

[0111] Specifically, in the case where it is determined according to the real-time evaluation results and the preset threshold that the first combination method is currently applicable, the audio and video data of all parties in the current live co-hosting are combined in the first combination method. The first combination method can be the local combination method or the server combination method. In the case where it is determined according to the real-time evaluation results and the preset threshold that the second combination method is currently applicable, switch from the first combination method to the second combination method, and combine the audio and video data of all parties in the current live co-hosting in the second combination method. Correspondingly, the second combination method can be the server combination method or the local combination method.

[0112] See Figure 5 shown, which is a schematic flowchart of another form of the live co-hosting combination switching method in this embodiment. Figure 5 The specific processes of each step have been described in detail above and will not be elaborated here.

[0113] The live co-hosting combination switching method proposed in this embodiment can analyze the hardware conditions and real-time network conditions of the host client based on dynamically configured evaluation rules during the live co-hosting process. Through the real-time evaluation results in each dimension, determine the most applicable co-hosting combination method currently, and dynamically switch different combination methods to adapt to the current live co-hosting scene, so as to improve the combination speed and effect, avoid problems such as freezing, black screen, and delay, and can also reduce the client pressure, ensure the co-hosting quality, thereby enhancing the host's co-hosting experience and the audience's viewing experience, and enhancing the attractiveness of the live broadcast room.

[0114] Embodiment 2

[0115] As Figure 6As shown in the figure, it is a schematic diagram of the hardware architecture of an electronic device 20 proposed in the second embodiment of the present application. In this embodiment, the electronic device 20 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23 that are communicatively connected to each other through a system bus. It should be noted that Figure 6 Only the electronic device 20 with components 21-23 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. In this embodiment, the electronic device 20 may be a host client, such as the first client.

[0116] The memory 21 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 21 may be an internal storage unit of the electronic device 20, such as the hard disk or memory of the electronic device 20. In other embodiments, the memory 21 may also be an external storage device of the electronic device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 20. Of course, the memory 21 may also include both the internal storage unit and the external storage device of the electronic device 20. In this embodiment, the memory 21 is generally used to store the operating system and various application software installed in the electronic device 20, such as the program code of the live connection and mixing switching system 60. In addition, the memory 21 may also be used to temporarily store various data that have been output or will be output.

[0117] In some embodiments, the processor 22 may be a CPU, a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 22 is generally used to control the overall operation of the electronic device 20. In this embodiment, the processor 22 is used to run the program code stored in the memory 21 or process data, such as running the live connection and mixing switching system 60, etc.

[0118] The network interface 23 may include a wireless network interface or a wired network interface, and this network interface 23 is generally used to establish a communication connection between the electronic device 20 and other electronic devices.

[0119] Embodiment Three

[0120] AsFigure 7 As shown in the figure, the following is a schematic diagram of the modules of a live connection and confluence switching system 60 proposed in the third embodiment of the present application. The live connection and confluence switching system 60 can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0121] In this embodiment, the live connection and confluence switching system 60 includes:

[0122] An acquisition module 600, configured to obtain hardware device evaluation parameters of the host client according to preset hardware influencing factors before the connection starts.

[0123] The acquisition module 600 first obtains the evaluation rules issued by the algorithm server. Before the connection starts, the acquisition module 600 obtains data and performs an evaluation on the local hardware device according to the evaluation rules according to the preset hardware influencing factors to obtain the hardware device evaluation parameters. Among them, the hardware influencing factors are hardware device factors that affect the connection quality, including model, CPU, graphics card, memory, etc. Since the data of each item of the hardware influencing factors are hardware dimension information, they can all be obtained through relevant API interfaces.

[0124] After obtaining the data corresponding to the hardware influencing factors, classification evaluations are respectively performed according to the evaluation rules, and the hardware device evaluation parameters are calculated. It should be noted that the weight corresponding to each influencing factor in the hardware influencing factors can be dynamically adjusted according to actual needs.

[0125] The acquisition module 600 is further configured to obtain real-time network evaluation parameters during the connection according to preset real-time influencing factors after the connection starts.

[0126] After the connection starts, the acquisition module 600 obtains data and performs an evaluation on the real-time network situation during the connection according to the evaluation rules according to the preset real-time influencing factors to obtain the real-time network evaluation parameters. Among them, the real-time influencing factors are network factors that affect the connection quality, including the main and guest network situations, push and pull stream frame rates, decoding and rendering situations, audio and video freezing rates, etc. The main and guest network situations can be obtained from the host client, and the push and pull stream frame rates, decoding and rendering situations, and audio and video freezing rates can be obtained from the connection SDK after the connection.

[0127] After obtaining the data corresponding to the real-time influencing factors, classification evaluation is also performed respectively according to the evaluation rules, and the real-time network evaluation parameters are calculated. The weights corresponding to each influencing factor in the real-time influencing factors can also be dynamically adjusted according to actual needs. Moreover, the real-time network evaluation parameters are obtained through real-time evaluation during the co-hosting process, that is to say, they will be continuously updated throughout the co-hosting process to subsequently determine the most applicable confluence method in real time.

[0128] The judgment module 602 is used to obtain a real-time evaluation result by combining the hardware device evaluation parameters and the real-time network evaluation parameters, and determine the currently applicable confluence method according to the real-time evaluation result and a preset threshold.

[0129] According to the hardware device evaluation parameters obtained before the co-hosting starts and the real-time network evaluation parameters obtained through real-time evaluation during the co-hosting process, combined calculation is performed according to the evaluation rules, and the final real-time evaluation result can be obtained. In this embodiment, the hardware device evaluation parameters and the real-time network evaluation parameters respectively correspond to dynamically adjustable weights. For example, the weight of the hardware device evaluation parameters is 60%, and the weight of the real-time network evaluation parameters is 40%. Multiplying the hardware device evaluation parameters and the real-time network evaluation parameters by their corresponding weights and then adding them together is the real-time evaluation result.

[0130] It should be noted that in an alternative embodiment, the weight of the real-time network evaluation parameters can also be set to 100%, and the weight of the hardware device evaluation parameters can be set to 0. That is to say, without considering the influence of the hardware device, the acquisition module 600 only obtains the real-time network evaluation parameters during the co-hosting process based on the real-time influencing factors, and the judgment module 602 obtains the real-time evaluation result according to the real-time network evaluation parameters.

[0131] Then, the judgment module 602 compares the real-time evaluation result with the preset threshold to determine the currently applicable confluence method. In this embodiment, the confluence methods include local confluence on the host client and server confluence. The confluence methods are divided according to the score of the real-time evaluation result. The case with a higher score is applicable to the local confluence method, which can guarantee the viewing experience of the audience-side users; while the case with a lower score is applicable to the server confluence method, which can reduce the pressure on the host client and improve the co-hosting quality. That is to say, when the real-time evaluation result is greater than the preset threshold, it is determined that the local confluence method is currently applicable; when the real-time evaluation result is less than or equal to the preset threshold, it is determined that the server confluence method is currently applicable.

[0132] The processing module 604 is used to perform audio and video confluence for the live co-hosting according to the determined confluence method.

[0133] After determining the currently applicable merging method based on the real-time evaluation result and the preset threshold, the determined merging method is used to merge the audio and video data of the current co-hosting, and push it to the CDN, so as to be distributed to each viewer terminal in the live broadcast room, so that users of the viewer terminal can watch the co-hosting scenario of the live broadcast room.

[0134] Since the real-time network evaluation parameters are evaluated and updated in real time during the co-hosting process, the real-time evaluation result will be updated continuously, which may cause the currently applicable merging method to change. When it is determined according to the real-time evaluation result that the currently applicable merging method has changed, it is necessary to switch from one merging method to another.

[0135] Specifically, in the case where the currently applicable first merging method is determined according to the real-time evaluation result and the preset threshold, the audio and video data of all parties in the current live co-hosting are merged in the first merging method. The first merging method can be the local merging method or the server merging method. In the case where the currently applicable second merging method is determined according to the real-time evaluation result and the preset threshold, switch from the first merging method to the second merging method, and merge the audio and video data of all parties in the current live co-hosting in the second merging method. Correspondingly, the second merging method can be the server merging method or the local merging method.

[0136] The live co-hosting merging switching system proposed in this embodiment can analyze the hardware conditions and real-time network conditions of the host client based on dynamically configured evaluation rules during the live co-hosting process, determine the most applicable co-hosting merging method currently through real-time evaluation results in each dimension, and dynamically switch different merging methods to adapt to the current live co-hosting scenario, so as to improve the merging speed and effect, avoid problems such as freezing, black screen, and delay, and can also reduce the client pressure, ensure the co-hosting quality, thereby enhancing the host co-hosting experience and the viewer viewing experience, and enhancing the attractiveness of the live broadcast room.

[0137] Embodiment 4

[0138] The present application also provides another implementation manner, that is, to provide a computer-readable storage medium, the computer-readable storage medium stores a live co-hosting merging switching program, and the live co-hosting merging switching program can be executed by at least one processor, so that the at least one processor executes the steps of the live co-hosting merging switching method as described above.

[0139] It should be noted that, in this text, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or apparatus that comprises a series of elements includes not only those elements but also other elements not expressly listed, or elements that are inherent to such process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or apparatus that comprises the element.

[0140] The serial numbers of the embodiments of the present application above are merely for description and do not represent the superiority or inferiority of the embodiments.

[0141] Obviously, those skilled in the art should understand that the various modules or steps of the embodiments of the present application described above can be implemented by a general-purpose computing device. They can be centralized on a single computing device or distributed across a network composed of multiple computing devices. Optionally, they can be implemented by program code executable by the computing device, so that they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order from here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module to be implemented. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0142] The above are only the preferred embodiments of the embodiments of the present application, and do not limit the patent scope of the embodiments of the present application. Any equivalent structural or equivalent process transformation made by using the specification and drawings of the embodiments of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the embodiments of the present application.

Claims

1. A live connection and merging switching method, which is applied to the host client, characterized in that, the method includes: After the connection starts, obtain the real-time network evaluation parameters during the connection according to the preset real-time influencing factors; Obtain the real-time evaluation result according to the real-time network evaluation parameters, and determine the currently applicable merging method according to the real-time evaluation result and the preset threshold; Perform audio and video merging of the live connection according to the determined merging method; Among them, the performing audio and video merging of the live connection according to the determined merging method includes: In the case where it is determined according to the real-time evaluation result and the preset threshold that the currently applicable first merging method is applicable, merge the audio and video data of the current live connection parties in the first merging method; In the case where it is determined according to the real-time evaluation result and the preset threshold that the currently applicable second merging method is applicable, switch from the first merging method to the second merging method, and merge the audio and video data of the current live connection parties in the second merging method; Among them, the method further includes: Before the connection starts, obtain the hardware device evaluation parameters of the host client according to the preset hardware influencing factors; and Obtain the real-time evaluation result by combining the hardware device evaluation parameters and the real-time network evaluation parameters.

2. The live connection and merging switching method according to claim 1, characterized in that, the merging methods include local merging on the host client and server merging.

3. The live connection and merging switching method according to claim 2, characterized in that, the local merging includes: Obtain the local screen and the screen of the other party in the connection; Merge the local screen and the other party's screen through texture rendering to obtain the connection scene screen; Obtain the local sound source and the other party's sound source; Process the local sound source and the other party's sound source through different audio tracks to obtain the connection audio; Push the connection scene screen and the connection audio to the content distribution network.

4. The live connection and merging switching method according to claim 2, characterized in that, the server merging includes: Obtain the local screen and the local sound source; Send the local screen and the local sound source to the connection server, so that the connection server merges the local screen and the screen of the other party in the connection to obtain the connection scene screen, merges the local sound source and the other party's sound source to obtain the connection audio, and pushes the connection scene screen and the connection audio to the content distribution network.

5. The live connection and merging switching method according to claim 2, characterized in that, the determining the currently applicable merging method according to the real-time evaluation result and the preset threshold includes: In the case where the real-time evaluation result is greater than the preset threshold, determine that the currently applicable local merging method is applicable; In the case where the real-time evaluation result is less than or equal to the preset threshold, determine that the currently applicable server merging method is applicable.

6. The live connection and merging switching method according to claim 1, characterized in that, The first-level weights for the dynamic adjustment corresponding to the hardware device evaluation parameters and the real-time network evaluation parameters. The first preset influencing factors include model, CPU, graphics card, and memory. The second preset influencing factors include the host-guest network situation, push-pull stream frame rate, decoding and rendering situation, and audio-video stuttering rate. Each influencing factor corresponds to a second-level weight for dynamic adjustment; Among them, the evaluation rules include the influencing factors on the co-hosting quality and the corresponding weights; The influencing factors include the first preset influencing factors and the second preset influencing factors. The first preset influencing factors correspond to the local hardware device situation of the client, and the second preset influencing factors correspond to the real-time network situation during the co-hosting process.

7. A live co-hosting and merging switching system, Characterized in that, The system includes: An acquisition module, configured to obtain real-time network evaluation parameters during the co-hosting process according to preset real-time influencing factors after the co-hosting starts; A judgment module, configured to obtain a real-time evaluation result according to the real-time network evaluation parameters, and determine the currently applicable merging method according to the real-time evaluation result and a preset threshold; A processing module, configured to perform audio-video merging of the live co-hosting according to the determined merging method, and push the merged data to the content delivery network; Among them, the performing audio-video merging of the live co-hosting according to the determined merging method includes: In the case where it is determined according to the real-time evaluation result and the preset threshold that the first merging method is currently applicable, merging the audio-video data of the current live co-hosting parties in the first merging method; In the case where it is determined according to the real-time evaluation result and the preset threshold that the second merging method is currently applicable, switching from the first merging method to the second merging method, and merging the audio-video data of the current live co-hosting parties in the second merging method; Among them, it further includes: The acquisition module is further configured to obtain hardware device evaluation parameters of the host client according to preset hardware influencing factors before the co-hosting starts; and The judgment module is further configured to obtain the real-time evaluation result by combining the hardware device evaluation parameters and the real-time network evaluation parameters.

8. An electronic device, Characterized in that, The electronic device includes: a memory, a processor, and a live co-hosting and merging switching program stored on the memory and executable on the processor. When the live co-hosting and merging switching program is executed by the processor, it implements the live co-hosting and merging switching method according to any one of claims 1 to 6.

9. A computer-readable storage medium, Characterized in that, A live co-hosting and merging switching program is stored on the computer-readable storage medium. When the live co-hosting and merging switching program is executed by a processor, it implements the live co-hosting and merging switching method according to any one of claims 1 to 6.

10. A computer program product, which includes computer instructions, Characterized in that, When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images