Picture switching method, device and equipment in SFU multi-party video conference

Abstract

The invention discloses a picture switching method, device and equipment in an SFU multi-party video conference. The method comprises the following steps: controlling a terminal, enabling the terminal to send video streams with at least two code rates, and staggering the sending time of I frames of the video streams with different code rates; mapping a display node for each display position, wherein a video stream of a terminal associated with the display node is placed on the corresponding display position; when a switching display request is received, switching the display node to a code rate stream which the frame I of the target terminal arrives at first; if the code rate stream which the I frame arrives at first is not the video stream with the maximum code rate, when the I frame of the video stream with the maximum code rate of the target terminal arrives, switching to the video stream with the maximum code rate; and otherwise, keeping the current code rate stream. According to the method, the corresponding video picture can be quickly switched, and the phenomenon of transient lagging or black screen at the display position is avoided.

Description

Technical Field

[0001] This invention relates to the technical field of video network transmission, and more specifically, to a method, apparatus, and device for switching screens in an SFU multi-party video conference. Background Technology

[0002] With the development of video conferencing technology, multi-view technology has emerged. In a video conference, each conference terminal can capture images and sound. After each conference terminal connects to the server and joins the conference, it encodes the captured images and sound and sends the encoded data to the server. The server then sends the images and sound sent by each conference terminal to other terminals according to the conference requirements, thereby enabling audio and video communication among multiple participants. This is the "one sender, multiple receivers" scenario of a conference terminal.

[0003] For SFU multi-party video conferencing systems, the timeliness of screen switching and the smoothness of the video feed are crucial evaluation metrics. To conserve bandwidth, terminals that are not currently displaying video streams are typically kept silent. If these terminals are to be switched onto the screen, and the forwarding source is immediately replaced, the I-frames sent by the terminals that are not currently displaying video either haven't arrived or have already passed (due to sending I-frame requests in advance), inevitably causing a brief stutter or black screen at the corresponding display location. Furthermore, in SFU's subscription-based video conferencing, different terminals can choose to watch different sources. If they need to switch displays, they must select the corresponding stream and place it at the appropriate location on the server. The traditional method is to immediately request an I-frame from the source upon switching. However, if multiple destinations are switching frequently, this can lead to a problem of simultaneously requesting I-frames from a single source, causing a short-term I-frame storm, resulting in network congestion and ultimately affecting the display quality. Summary of the Invention

[0004] To overcome at least one of the defects described in the prior art, the present invention provides a method, apparatus, and device for switching screens in SFU multi-party video conferencing. The technical solution adopted by the present invention is as follows.

[0005] In a first aspect, the present invention provides a method for switching screens in an SFU (Simultaneous Multi-Party Video Conferencing) multi-party video conference, the method being applied to a conference server in an SFU multi-party video conference, comprising:

[0006] The control terminal sends at least two video streams with different bitrates, and the I-frames of the different bitrate video streams are sent at different times, so that the arrival times of the I-frames of the different bitrate video streams are dispersed; among them, the video stream with the highest bitrate is the video stream that is finally placed on the display position of the video conference.

[0007] For each display location, a display node is mapped, and the video stream of the terminal associated with the display node is placed in the corresponding display location;

[0008] When a display switching request is received, the display node is associated with the terminal to be switched according to the display switching request. The terminal to be switched is the target terminal. The arrival time of the I-frame of the video stream with different bitrates of the target terminal is calculated, and the display node is switched to the bitrate stream with the first I-frame to arrive.

[0009] If the first bitrate stream to arrive at the I-frame is not the video stream with the highest bitrate, wait until the I-frame of the video stream with the highest bitrate arrives at the target terminal, and then switch to the video stream with the highest bitrate.

[0010] If the first bitrate stream to arrive in the I-frame is the video stream with the highest bitrate, then maintain the current bitrate stream.

[0011] In one implementation, for at least two video streams with different bitrates sent by the terminal, if N video streams with different bitrates are sent, and the length of the GOP for each video stream with different bitrates is S seconds, then the duration of the I-frame interval between each video stream with different bitrates is (S / N) seconds.

[0012] In one embodiment, the screen switching method in the SFU multi-party video conference further includes:

[0013] When the conference server receives a switching display request initiated by the destination, but the bitrate stream of the first I-frame of the target terminal has not yet arrived, the display node maintains the video stream of the original display node until the bitrate stream of the first I-frame of the target terminal arrives.

[0014] In one embodiment, the screen switching method in the SFU multi-party video conference further includes:

[0015] The conference server establishes data and control channels for each terminal;

[0016] In the control channel, the conference server establishes channel nodes for the audio and video streams of all terminals. Each channel node corresponds to an audio and video stream channel, and the display status of the channel node is marked as ON or OFF. The ON state indicates that the audio and video stream is forwarded to other terminals, and the OFF state indicates that the audio and video stream is discarded.

[0017] In one embodiment, the screen switching method in the SFU multi-party video conference further includes:

[0018] The conference server establishes data and control channels for each terminal;

[0019] In the control channel, the conference server establishes channel nodes for the audio and video streams of all terminals. Each channel node corresponds to an audio and video stream channel, and the display status of the channel node is marked as ON or OFF. The ON state indicates that the audio and video stream is forwarded to other terminals, and the OFF state indicates that the audio and video stream is discarded.

[0020] When the conference server receives a switch display request from the destination, it changes the status of the audio and video stream that is about to be switched to the waiting OFF state.

[0021] When the audio / video stream to be switched out is switched out, the state of the audio / video stream to be switched out is changed to OFF state.

[0022] In one embodiment, the screen switching method in the SFU multi-party video conference further includes:

[0023] When the conference server receives a switch display request from the destination, it switches the status of the target terminal's audio and video streams to the first waiting ON state.

[0024] When the target terminal's audio and video stream is switched to a video stream with the highest bitrate, the status of the target terminal's audio and video stream is changed to the second waiting ON state.

[0025] When the target terminal is switched to the video stream with the highest bitrate, the status of the target terminal's audio and video streams is switched to ON.

[0026] Secondly, the present invention provides a screen switching device for SFU multi-party video conferencing, the device being applied to a conference server in SFU multi-party video conferencing, comprising:

[0027] The terminal control module is used to control the terminal to send at least two video streams with different bitrates, and the transmission time of the I-frames of the different bitrate video streams is staggered, so that the arrival time of the I-frames of the different bitrate video streams is dispersed; among them, the video stream with the highest bitrate is the video stream that is finally placed in the display position of the video conference.

[0028] The association module is used to map display nodes to each display location, and the video stream of the terminal associated with the display node will be placed in the corresponding display location;

[0029] The switching module is used to, upon receiving a display switching request, associate the display node with the terminal to be switched (i.e., the target terminal) according to the request. It calculates the arrival times of I-frames of different bitrate video streams at the target terminal and switches the display node to the bitrate stream whose I-frame arrives first. If the bitrate stream whose I-frame arrives first is not the video stream with the highest bitrate, it waits for the I-frame of the video stream with the highest bitrate at the target terminal to arrive and then switches to the video stream with the highest bitrate. If the bitrate stream whose I-frame arrives first is the video stream with the highest bitrate, it maintains the current bitrate stream.

[0030] In one implementation, for at least two video streams with different bitrates sent by the terminal, if N video streams with different bitrates are sent, and the length of the GOP for each video stream with different bitrates is S seconds, then the duration of the I-frame interval between each video stream with different bitrates is (S / N) seconds.

[0031] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any of the above embodiments.

[0032] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method of any of the above embodiments.

[0033] In this invention, during SFU multi-party video conferencing, each terminal sends video streams with multiple bitrates, and the I-frames of each video stream are staggered. When switching between views, the system first switches to the video stream with the fastest arriving I-frame, and then switches to the final required bitrate video stream. This method enables rapid switching to the corresponding video view, avoiding brief stutters or black screens. Furthermore, this invention facilitates the separation of audio / video channels and signaling channels by the conferencing server. The signaling channel only handles setting the status, while the audio / video channels only handle converting audio and video streams based on the status and frame type, transitioning to the target stream as quickly as possible without requesting I-frames. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall process of one embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of the state change of a channel node that is about to exit the display in Embodiment 1 of the present invention.

[0036] Figure 3 This is a schematic diagram of the state changes of a channel node that is about to be displayed in Embodiment 1 of the present invention.

[0037] Figure 4This is a schematic diagram of the overall structure of Embodiment 2 of the present invention. Detailed Implementation

[0038] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0039] It should be noted that the terms "first, second, ..." used in the embodiments of the present invention are merely used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, ..." can be interchanged in a specific order or sequence where permissible. It should be understood that the objects distinguished by "first, second, ..." can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0040] Example 1

[0041] Please see Figure 1 , Figure 1 This is a flowchart illustrating a screen switching method in an SFU (Simultaneous Multi-Party Video Conferencing) multi-party video conference according to Embodiment 1 of the present invention. The method is applied to a conference server in an SFU multi-party video conference and includes steps S110, S120, S130, S140, and S150. It should be noted that steps S110, S120, S130, S140, and S150 are merely reference numerals used to clearly explain the embodiment and the accompanying drawings. Figure 1 The correspondence is not intended to limit the order of steps in this embodiment.

[0042] Step S110: Control the terminal to send at least two video streams with different bitrates, and stagger the sending times of the I-frames of the different bitrate video streams so that the arrival times of the I-frames of the different bitrate video streams are dispersed; among them, the video stream with the highest bitrate is the video stream that is finally placed on the display position of the video conference.

[0043] Step S120: Map a display node for each display position, and the video stream of the terminal associated with the display node will be placed at the corresponding display position;

[0044] Step S130: When a display switching request is received, the display node is associated with the terminal to be switched according to the display switching request. The terminal to be switched is the target terminal. The arrival time of the I-frame of the video stream with different bitrates of the target terminal is calculated, and the display node is switched to the bitrate stream with the first I-frame to arrive.

[0045] Step S140: If the first bitrate stream to arrive at the I-frame is not the video stream with the highest bitrate, wait for the I-frame of the video stream with the highest bitrate to arrive at the target terminal, and then switch to the video stream with the highest bitrate.

[0046] Step S150: If the first bitrate stream to arrive in the I-frame is the video stream with the highest bitrate, maintain the current bitrate stream.

[0047] In video conferencing, each conference terminal is connected to the conference server. This method is used to control the operation of the conference server and can be applied to the server itself or to the device that controls the server.

[0048] A multi-party SFU video conference involves terminals establishing a data channel with the conference server. The terminals send audio and video streams to the server through this channel. These terminals can periodically transmit at least two video streams with different bitrates, such as high, medium, and low bitrate streams. The transmission times of the I-frames for each bitrate stream are staggered, resulting in dispersed arrival times for the I-frames. For example, if the Group of Optical Pages (GOPs) is set to 4 seconds, the interval between the first frames of the three streams is set to 1.33 seconds. Although these terminals can transmit at least two bitrate video streams at even intervals, the stream with the highest bitrate—the high-bitrate stream in the example above—is ultimately displayed on the screen.

[0049] It should be noted that, preferably, all terminals can transmit video streams at at least two bitrates at intervals. However, depending on the actual situation, it is also possible that only some terminals can transmit video streams at at least two bitrates at intervals. In this case, steps S120-S150 are only executed on the video streams transmitted by the terminals that can transmit video streams at at least two bitrates at intervals.

[0050] At the start of the meeting, the meeting server automatically allocates limited display positions to the corresponding terminals according to the order of joining. Then, the audio and video stream nodes of the terminals that have obtained display positions are placed on the corresponding display nodes. The display nodes then forward the audio and video streams from the terminals, so that other terminals can see the screen of the terminal in the corresponding display position.

[0051] The conferencing server simulates a real-world conference by establishing multiple display nodes. Each display node forwards audio and video streams from only one terminal at a time, but a single terminal can send streams at multiple bitrates. This single-node forwarding of audio and video streams from a single terminal facilitates the expansion and management of audio and video streams in video conferencing. Allowing a single terminal to send streams at multiple bitrates allows the server to send different types of streams to different terminals in real time based on downlink bandwidth conditions. Furthermore, it provides a trade-off between smoothness and real-time performance when switching displays.

[0052] The conferencing server continuously monitors the arrival times of I-frames for various streams in each display node. When a destination needs to switch the stream of a certain screen, the destination sends a display switching request to the conferencing server. Upon receiving the request, the server associates the display node with the terminal to be switched, which is the target terminal. Then, it calculates the stream type of the first I-frame to arrive at the corresponding display node and displays that stream at the corresponding location. For example, if the fastest arriving I-frame is a medium stream I-frame, which will arrive in 0.2 seconds, while the large stream I-frame will arrive in 1.52 seconds, then it is set to switch to the medium stream first. Of course, if the fastest arriving I-frame is a large stream I-frame, then it is set to switch to the large stream first.

[0053] As mentioned earlier, the video stream with the highest bitrate is ultimately displayed in the video conference. Therefore, if you switch to the intermediate stream first, as in the example above, you must wait for the I-frame of the main stream to arrive before switching back to the main stream. If the fastest arriving I-frame is the I-frame of the main stream, then you can simply maintain the switch since you have already switched to the main stream.

[0054] In existing technologies, each terminal generates only one video stream, meaning only one I-frame is generated within the duration of a Group of Pictures (GOP). When switching scenes, the system must wait for this I-frame to arrive, which is very time-consuming and prone to stuttering and black screens. This method, by generating video streams with multiple bitrates and staggered I-frames, ensures that there are multiple I-frames within the duration of a GOP. When switching scenes, it only needs to wait for the nearest I-frame, saving time and reducing stuttering.

[0055] In this method, during SFU multi-party video conferencing, each terminal sends video streams with multiple bitrates, and the I-frames of each video stream are staggered. When switching between views, the system first switches to the video stream with the fastest arriving I-frame, and then switches to the final required bitrate video stream. This method enables rapid switching to the corresponding video view, avoiding brief stutters or black screens.

[0056] In one implementation, for at least two video streams with different bitrates sent by the terminal, if N video streams with different bitrates are sent, and the length of the GOP for each video stream with different bitrates is S seconds, then the duration of the I-frame interval between each video stream with different bitrates is (S / N) seconds.

[0057] This is the scenario described earlier where various bitrates are transmitted at uniform intervals. For example, if there are three different bitrate streams (high, medium, and low) as mentioned earlier, and the GOP is set to 4 seconds, then the interval between the first frame transmissions of the three streams is set to 1.33 seconds.

[0058] It should be noted that this implementation method only addresses how to set the I-frame interval for each terminal. However, in reality, the value of N varies between terminals. Some terminals may be divided into three types: large, medium, and small, while others may be divided into four types: large, medium, medium, and small. The different values ​​of N between terminals will result in different I-frame intervals for each terminal.

[0059] In one embodiment, the screen switching method in the SFU multi-party video conference further includes step S160.

[0060] In step S160, when the conference server receives a switching display request initiated by the destination, but the bitrate stream of the first I-frame of the target terminal has not arrived, the display node maintains the video stream of the original display node until the bitrate stream of the first I-frame of the target terminal arrives.

[0061] If the conference server switches immediately upon receiving a switch request from the destination, there will be no video output and a brief black screen because the bitrate stream of the first I-frame of the target terminal has not yet arrived. Therefore, this implementation allows the display node to maintain the original video stream of the display node for a period of time until the bitrate stream of the first I-frame of the target terminal arrives before switching, thereby avoiding the occurrence of a black screen.

[0062] In one embodiment, the screen switching method in the SFU multi-party video conference further includes steps S210 and S220.

[0063] Step S210: The conference server establishes a data channel and a control channel for each terminal;

[0064] In step S220, in the control channel, the conference server establishes channel nodes for the audio and video streams of all terminals. Each channel node corresponds to an audio and video stream channel, and the display status of the channel node is marked as ON or OFF. The ON state indicates that the audio and video stream is forwarded to other terminals, and the OFF state indicates that the audio and video stream is discarded.

[0065] The conferencing server establishes channel nodes for all audio and video streams (one node corresponds to one data channel). It then classifies the channel node status of the source audio and video streams from all terminals into two states: OFF and ON. At any given time on a display node, there is exactly one source terminal's audio and video stream channel node, and that channel node is marked as ON. The conferencing server forwards the audio and video streams received from the corresponding terminal and marked as ON to other terminals. For terminals not included in the display node, the audio and video stream nodes are marked as OFF, and any audio and video streams received from the corresponding terminal and marked as OFF are discarded.

[0066] It should be noted that in this embodiment, the term "channel node" refers to a unit that manages the uploaded audio and video streams of the server's management terminal. In contrast, the term "display node" was used previously. A display node is a virtual unit created by the server based on the actual screen display position. It is used to forward audio and video streams from the channel node and also has the function of managing the downlink channel.

[0067] The conferencing server establishes a corresponding stream node for each audio and video channel. At any given time, the conferencing server will only place one stream node in the display node and provide forwarding services. This allows for the creation of nodes for audio and video streams, facilitating the entry and exit of different audio and video streams into different display nodes and simplifying the management of audio and video stream forwarding services.

[0068] In one embodiment, the screen switching method in the SFU multi-party video conference further includes steps S230 and S240.

[0069] Step S230: When the conference server receives a switch display request initiated by the destination, it changes the status of the audio and video stream that is about to be switched to the waiting OFF state.

[0070] Step S240: When the audio / video stream to be switched out is switched out, the state of the audio / video stream to be switched out is changed to OFF state.

[0071] like Figure 2 As shown, Figure 2 This diagram illustrates the state changes of a channel node that is about to be removed from the display. When the conferencing server receives a display switch request from the destination, but the first bitrate stream of the target terminal's I-frame has not yet arrived (meaning the forwarded stream is still the audio / video stream to be switched out), the state of the audio / video stream to be switched out is changed to the waiting OFF state. Once the first bitrate stream of the target terminal's I-frame arrives, the audio / video stream to be switched out will be switched out, and its state will then change to the OFF state.

[0072] In one embodiment, the screen switching method in the SFU multi-party video conference further includes steps S250, S260 and S270.

[0073] Step S250: When the conference server receives a switch display request initiated by the destination, it switches the status of the audio and video stream of the target terminal to the first waiting ON state.

[0074] Step S260: When the target terminal's audio and video stream is switched to a video stream with the highest bit rate but not the target terminal's audio and video stream, the target terminal's audio and video stream status is changed to the second waiting ON state.

[0075] Step S270: When the target terminal is switched to the video stream with the highest bitrate, the status of the target terminal's audio and video streams is switched to ON.

[0076] like Figure 3 As shown, Figure 3 This diagram illustrates the state changes of a channel node about to be displayed. When the conference server receives a display switching request from the destination, but the first bitrate stream of the target terminal's I-frame has not yet arrived (meaning the forwarded stream is still the audio / video stream to be switched out), the state of the target terminal's audio / video stream is changed to the first waiting ON state. When the first bitrate stream of the target terminal's I-frame arrives, two scenarios can occur: either the intermediate stream (not the highest bitrate video stream) or the highest bitrate video stream arrives. If the intermediate stream arrives, the state of the target terminal's audio / video stream is changed to the second waiting ON state, and it remains there until the highest bitrate video stream arrives before switching to the ON state; if the highest bitrate video stream arrives, it switches directly to the ON state.

[0077] This allows the conferencing server to separate the audio / video channels from the signaling channels. The signaling channel only needs to set the status, while the audio / video channels only need to convert the audio / video streams according to the status and frame type, transitioning to the target stream as quickly as possible without requesting I-frames.

[0078] Example 2

[0079] Corresponding to the method in Example 1, such as Figure 4 As shown, the present invention also provides a screen switching device 4 for SFU multi-party video conferencing. The device is applied to the conference server in SFU multi-party video conferencing and includes: a terminal control module 410, an association module 420, and a switching module 430.

[0080] The terminal control module 410 is used to control the terminal to send at least two video streams with different bitrates, and the transmission time of the I-frames of the different bitrate video streams is staggered, so that the arrival time of the I-frames of the different bitrate video streams is dispersed; among them, the video stream with the highest bitrate is the video stream that is finally placed in the display position of the video conference.

[0081] The association module 420 is used to map display nodes for each display position, and the video stream of the terminal associated with the display node will be placed in the corresponding display position;

[0082] The switching module 430 is used to, when receiving a display switching request, associate the display node with the terminal to be switched, which is the target terminal, calculate the arrival time of I-frames of different bitrate video streams of the target terminal, and switch the display node to the bitrate stream whose I-frame arrives first; if the bitrate stream whose I-frame arrives first is not the video stream with the highest bitrate, wait for the I-frame of the video stream with the highest bitrate of the target terminal to arrive, and switch to the video stream with the highest bitrate; if the bitrate stream whose I-frame arrives first is the video stream with the highest bitrate, maintain the current bitrate stream.

[0083] In one implementation, for at least two video streams with different bitrates sent by the terminal, if N video streams with different bitrates are sent, and the length of the GOP for each video stream with different bitrates is S seconds, then the duration of the I-frame interval between each video stream with different bitrates is (S / N) seconds.

[0084] In one implementation, the switching module is further configured to, when the conference server receives a switching display request initiated by the destination, but the bitrate stream of the first I-frame of the target terminal has not yet arrived, maintain the video stream of the original display node until the bitrate stream of the first I-frame of the target terminal arrives.

[0085] In one implementation, the terminal control module is further configured to establish a data channel and a control channel for each terminal; in the control channel, the conference server establishes channel nodes for the audio and video streams of all terminals, each channel node corresponds to an audio and video stream channel, and marks the display status of the channel node as ON or OFF, wherein the ON state indicates that the audio and video stream is forwarded to other terminals, and the OFF state indicates that the audio and video stream is discarded.

[0086] In one implementation, the terminal control module is further configured to establish a data channel and a control channel for each terminal. In the control channel, the conference server establishes channel nodes for the audio and video streams of all terminals. Each channel node corresponds to an audio and video stream channel, and the display status of the channel node is marked as ON or OFF. The ON state indicates that the audio and video stream is forwarded to other terminals, and the OFF state indicates that the audio and video stream is discarded. When the conference server receives a switching display request initiated by the destination terminal, it changes the status of the audio and video stream that is to be switched to the waiting OFF state. When the audio and video stream that is to be switched is switched, the status of the switched audio and video stream is changed to the OFF state.

[0087] In one embodiment, the terminal control module is further configured to switch the status of the audio and video stream of the target terminal to the first waiting ON state when the conference server receives a switching display request initiated by the target terminal.

[0088] When the target terminal's audio and video stream is switched to a video stream with the highest bitrate, the status of the target terminal's audio and video stream is changed to the second waiting ON state.

[0089] When the target terminal is switched to the video stream with the highest bitrate, the status of the target terminal's audio and video streams is switched to ON.

[0090] In this device, during SFU multi-party video conferencing, each terminal sends video streams at multiple bitrates, with staggered I-frames for each stream. When switching between streams, the system first switches to the stream with the fastest arriving I-frame, and then switches to the desired bitrate. This method allows for rapid switching to the corresponding video feed, avoiding brief stutters or black screens. Furthermore, this device facilitates the separation of audio / video and signaling channels by the conferencing server. The signaling channel only handles setting the status, while the audio / video channel only converts the audio and video streams based on the status and frame type, transitioning to the target stream as quickly as possible without requesting I-frames.

[0091] Example 3

[0092] This invention also provides a storage medium storing computer instructions that, when executed by a processor, implement the screen switching method in SFU multi-party video conferencing as described above.

[0093] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, random access memory (RAM), read-only memory (ROM), magnetic disks, or optical disks.

[0094] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, RAM, ROM, magnetic disks, or optical disks.

[0095] Corresponding to the computer storage medium described above, one embodiment also provides a computer device, which includes a memory, an encoder, and a computer program stored in the memory and executable on the encoder, wherein the encoder executes the program to implement any of the screen switching methods in SFU multi-party video conferencing as described in the above embodiments.

[0096] In SFU multi-party video conferencing, the aforementioned computer equipment allows each terminal to send video streams at multiple bitrates. The I-frames of each video stream are staggered. When switching between views, the system first switches to the video stream with the fastest arriving I-frame, and then switches to the final required bitrate video stream. This method enables rapid switching to the corresponding video view, avoiding brief stutters or black screens. Furthermore, this computer equipment facilitates the separation of audio / video channels and signaling channels by the conferencing server. The signaling channel only handles setting the status, while the audio / video channels only handle converting audio and video streams based on the status and frame type, transitioning to the target stream as quickly as possible without requesting I-frames.

[0097] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for switching screens in an SFU multi-party video conference, wherein the method is applied to a conference server in an SFU multi-party video conference, characterized in that, include: The control terminal sends at least two video streams with different bitrates, and the sending times of the I-frames of the different bitrate video streams are staggered, so that the arrival times of the I-frames of the different bitrate video streams are dispersed. The video stream with the highest bitrate is the one that will ultimately be displayed in the video conference. For each display location, a display node is mapped, and the video stream of the terminal associated with the display node is placed in the corresponding display location; When a display switching request is received, the display node is associated with the terminal to be switched according to the display switching request. The terminal to be switched is the target terminal. The arrival time of the I-frame of the video stream with different bitrates of the target terminal is calculated, and the display node is switched to the bitrate stream with the first I-frame to arrive. If the first bitrate stream to arrive at the I-frame is not the video stream with the highest bitrate, wait until the I-frame of the video stream with the highest bitrate arrives at the target terminal, and then switch to the video stream with the highest bitrate. If the first bitrate stream to arrive in the I-frame is the video stream with the highest bitrate, then maintain the current bitrate stream.

2. The screen switching method in SFU multi-party video conferencing according to claim 1, characterized in that, For a terminal sending at least two video streams with different bitrates, if the number of video streams sent is N and the length of the GOP for each bitrate video stream is S seconds, then the duration of the I-frame interval between each bitrate video stream is (S / N) seconds.

3. The screen switching method in SFU multi-party video conferencing according to claim 1, characterized in that, Also includes: When the conference server receives a switching display request initiated by the destination, but the bitrate stream of the first I-frame of the target terminal has not yet arrived, the display node maintains the video stream of the original display node until the bitrate stream of the first I-frame of the target terminal arrives.

4. The screen switching method in SFU multi-party video conferencing according to claim 1, characterized in that, Also includes: The conference server establishes data and control channels for each terminal; In the control channel, the conference server establishes channel nodes for the audio and video streams of all terminals. Each channel node corresponds to an audio and video stream channel, and the display status of the channel node is marked as ON or OFF. The ON state indicates that the audio and video stream is forwarded to other terminals, and the OFF state indicates that the audio and video stream is discarded.

5. The screen switching method in SFU multi-party video conferencing according to claim 3, characterized in that, Also includes: The conference server establishes data and control channels for each terminal; In the control channel, the conference server establishes channel nodes for the audio and video streams of all terminals. Each channel node corresponds to an audio and video stream channel, and the display status of the channel node is marked as ON or OFF. The ON state indicates that the audio and video stream is forwarded to other terminals, and the OFF state indicates that the audio and video stream is discarded. When the conference server receives a switch display request from the destination, it changes the status of the audio and video stream that is about to be switched to the waiting OFF state. When the audio / video stream to be switched out is switched out, the state of the audio / video stream to be switched out is changed to OFF state.

6. The screen switching method in SFU multi-party video conferencing according to claim 5, characterized in that, Also includes: When the conference server receives a switch display request from the destination, it switches the status of the target terminal's audio and video streams to the first waiting ON state. When the target terminal's audio and video stream is switched to a video stream with the highest bitrate, the status of the target terminal's audio and video stream is changed to the second waiting ON state. When the target terminal is switched to the video stream with the highest bitrate, the status of the target terminal's audio and video streams is switched to ON.

7. A screen switching device for SFU multi-party video conferencing, the device being applied to a conference server in SFU multi-party video conferencing, characterized in that, include: The terminal control module is used to control the terminal to send at least two video streams with different bitrates, and the sending time of the I-frames of the different bitrate video streams is staggered, so that the arrival time of the I-frames of the different bitrate video streams is dispersed. The video stream with the highest bitrate is the one that will ultimately be displayed in the video conference. The association module is used to map display nodes to each display location, and the video stream of the terminal associated with the display node will be placed in the corresponding display location; The switching module is used to, upon receiving a display switching request, associate the display node with the terminal to be switched (i.e., the target terminal) according to the request. It calculates the arrival times of I-frames of different bitrate video streams at the target terminal and switches the display node to the bitrate stream whose I-frame arrives first. If the bitrate stream whose I-frame arrives first is not the video stream with the highest bitrate, it waits for the I-frame of the video stream with the highest bitrate at the target terminal to arrive and then switches to the video stream with the highest bitrate. If the bitrate stream whose I-frame arrives first is the video stream with the highest bitrate, it maintains the current bitrate stream.

8. The screen switching device in SFU multi-party video conferencing according to claim 7, characterized in that, For a terminal sending at least two video streams with different bitrates, if the number of video streams sent is N and the length of the GOP for each bitrate video stream is S seconds, then the duration of the I-frame interval between each bitrate video stream is (S / N) seconds.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images