Control method, multipoint control unit and storage medium of video conference system

Abstract

The present invention discloses a control method, a multipoint control unit and a storage medium for a video conference system, wherein the control method for the video conference system is applied to the multipoint control unit, and comprises the following steps: in response to a private chat conference request initiated by a first video conference terminal to a second video conference terminal, the code stream of the second video conference terminal is obtained through the second multipoint control unit, and the second video conference terminal is connected to the second multipoint control unit; the code streams of the second video conference terminal and the third video conference terminal are sent to the first video conference terminal; the code stream of the first video conference terminal is sent to the second video conference terminal through the second multipoint control unit. Thus, the first video conference terminal and the second video conference terminal are able to establish a private chat sub-conference under a multi-level video conference, and the audio and video data of the private chat sub-conference will not affect the progress of the main venue, which brings convenience to the application of video conferences.

Description

Technical Field

[0001] The present invention relates to the technical field of video conferencing, and in particular to a control method, a multipoint control unit and a video conferencing terminal of a video conferencing system. Background Art

[0002] Video conferencing is a real-time, two-way or multi-party video communication conference held through video conferencing terminals and video conferencing multipoint control units (MCU). Through video conferencing, people can conveniently conduct remote meetings, training and teaching in multiple locations.

[0003] Generally speaking, a video conference includes a main conference terminal that serves as a broadcast source and a video conference terminal that receives the broadcast source. The video conference terminal and the main conference terminal transmit code streams through video channels. During a video conference, when participants need to communicate with each other, they need to establish a private chat sub-conference in the current video conference. However, under current technology, it is only possible to establish a private chat sub-conference in a single-level conference, that is, the main conference terminal that serves as a broadcast source and the two video conference terminals that establish the private chat sub-conference need to be directly connected to the same multi-point control unit, and the multi-point control unit establishes the private chat sub-conference. However, in the case of cascading more than two multi-point control units, a private chat sub-conference cannot be established. Summary of the invention

[0004] The main purpose of the embodiment of the present invention is to provide a control method for a video conferencing system, aiming to establish a private chat sub-conference in a multi-level video conference.

[0005] To achieve the above object, an embodiment of the present invention provides a method for controlling a video conferencing system, comprising:

[0006] The video conferencing system includes a first video conferencing terminal, a second video conferencing terminal, a third video conferencing terminal, a first multipoint control unit and a second multipoint control unit. The control method is applied to the first multipoint control unit, and the control method includes:

[0007] Establishing connections with the first video conferencing terminal and the third video conferencing terminal respectively, and receiving code streams from the first video conferencing terminal and the third video conferencing terminal;

[0008] In response to a private chat conference request initiated by the first video conferencing terminal to the second video conferencing terminal, acquiring a code stream of the second video conferencing terminal through the second multipoint control unit, wherein the second video conferencing terminal is connected to the second multipoint control unit;

[0009] Sending code streams of the second video conferencing terminal and the third video conferencing terminal to the first video conferencing terminal;

[0010] The code stream of the first video conferencing terminal is sent to the second video conferencing terminal through the second multipoint control unit.

[0011] In order to achieve the above objectives, an embodiment of the present invention also provides a multi-point control unit, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the aforementioned control method of the video conferencing system when executed by the processor.

[0012] In order to achieve the above-mentioned purpose, an embodiment of the present invention further provides a computer-readable storage medium, on which an information processing program is stored, and when the information processing program is executed by a processor, the control method of the above-mentioned video conferencing system is implemented.

[0013] The embodiment of the present invention proposes a control method, a multipoint control unit and a storage medium for a video conferencing system. The control method of the video conferencing system is used to establish a private chat sub-conference for two video conferencing terminals under the current multi-level video conferencing. For example, under the current video conferencing, the first video conferencing terminal is directly connected to the first multipoint control unit, and the second video conferencing terminal is directly connected to the second multipoint control unit. The first multipoint control unit and the second multipoint control unit are cascaded or belong to two different conference venues. When the first multipoint control unit receives a private chat conference request initiated by the first video conferencing terminal to the second video conferencing terminal, the first multipoint control unit creates a new sub-conference for the first video conferencing terminal and the second video conferencing terminal, and controls the exchange of audio and video data between the first video conferencing terminal and the second video conferencing terminal, and at the same time provides the first video conferencing terminal with audio and video data of the third video conferencing terminal as a broadcast source. Thus, the first video conferencing terminal and the second video conferencing terminal establish a private chat sub-conference under the multi-level video conferencing, and the audio and video data of the private chat sub-conference will not affect the third video conferencing terminal as a broadcast source, which brings convenience to the application of video conferencing. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a schematic diagram of a connection structure of a video conferencing system with a cascade structure provided by an embodiment of the present invention;

[0015] Figure 2 is a flow chart of a control method for a video conferencing system provided by an embodiment of the present invention;

[0016] Figure 3 is a flow chart of a control method of a video conferencing system provided by another embodiment of the present invention;

[0017] Figure 4 is a flow chart of a control method of a video conferencing system provided by another embodiment of the present invention;

[0018] Figure 5 is a schematic diagram of a connection structure of a video conferencing system with a cascade structure provided by another embodiment of the present invention;

[0019] Figure 6 is a flow chart of a control method of a video conferencing system provided by another embodiment of the present invention;

[0020] Figure 7 is a schematic diagram of a connection structure of a video conferencing system including two main conference sites provided by another embodiment of the present invention;

[0021] Figure 8 is a flow chart of a control method of a video conferencing system provided by another embodiment of the present invention;

[0022] Fig. 9 is a flow chart of a control method of a video conferencing system provided by another embodiment of the present invention;

[0023] Fig.10 is a flow chart of a control method of a video conferencing system provided by another embodiment of the present invention;

[0024] Fig.11 It is a device connection structure diagram of a multi-point control unit provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0026] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no special meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0027] In a video conference, a multipoint control unit (MCU) selects a video conference terminal as the main venue terminal, receives the audio and video stream of the video conference terminal and broadcasts it to other video conference terminals, and the audio and video streams of other video conference terminals are also broadcasted by the multipoint control unit; during the video conference, the participants (video conference terminals) may need to discuss with each other. In order not to affect the normal progress of the main venue, the multipoint control unit often needs to disconnect the video conference terminal for private chat and exchange audio and video data through the multipoint control unit. This requires that the two video conference terminals for private chat are directly connected to the same multipoint control unit for control. With the development of multi-level conferences, there are more and more cases of cascading meetings of multiple video conference terminals. However, it is impossible to establish a private chat sub-conference under the cascade structure, which brings inconvenience to the application of video conferences. Based on this, the following embodiment of the present invention provides a method for establishing a private chat sub-conference under a multi-level video conference.

[0028] The present embodiment provides a control method for a video conferencing system, which is a method for establishing a private chat sub-conference in a multi-level video conference. The multi-level video conference has more than two multi-point control units, and two or more video conferencing terminals for establishing private chats are directly connected to at least two different multi-point control units. To simplify the description, the control method of the embodiment of the present invention is described below using two multi-point control units and two video conferencing terminals as an example.

[0029] In one embodiment, referring to Figure 1 The video conferencing system has a cascade structure, which is composed as follows: the first multi-point control unit is directly connected to the first video conferencing terminal (represented by terminal A in the figure), the second multi-point control unit is directly connected to the second video conferencing terminal (represented by terminal B in the figure), and the first multi-point control unit is connected to the second multi-point control unit to form a cascade conference, which also includes a third video conferencing terminal (represented by terminal C in the figure) directly connected to the first multi-point control unit. The third video conferencing terminal serves as the main venue, and the first multi-point control unit broadcasts the code stream of the third video conferencing terminal to each of the following video conferencing terminals.

[0030] like Figure 2 As shown, Figure 2 is a flow chart of a control method of the video conferencing system in the above embodiment, the control method is applied to the first multipoint control unit, and comprises the following steps:

[0031] S110: Establishing connections with the first video conferencing terminal and the third video conferencing terminal respectively, and receiving code streams from the first video conferencing terminal and the third video conferencing terminal;

[0032] In this embodiment, the first multi-point control unit is directly connected to the third video conferencing terminal and the third video conferencing terminal is selected as the main venue. Since the second multi-point control unit is connected to the first multi-point control unit to form a cascade conference, from the perspective of the first multi-point control unit, the first multi-point control unit needs to receive the code streams of the first video conferencing terminal and the third video conferencing terminal and broadcast them so that the second video conferencing terminal can view the images of the first video conferencing terminal and the third video conferencing terminal. At the same time, it also receives the code stream of the second video conferencing terminal sent by the second multi-point control unit and broadcasts it.

[0033] S120, in response to a private chat conference request initiated by the first video conferencing terminal to the second video conferencing terminal, acquiring a code stream of the second video conferencing terminal through the second multipoint control unit;

[0034] When the first video conferencing terminal needs to have a private chat with the second video conferencing terminal, the first video conferencing terminal sends a private chat conference request to the first multi-point control unit. After receiving the private chat conference request initiated by the first video conferencing terminal, the first multi-point control unit establishes a private chat sub-conference for the first video conferencing terminal and the second video conferencing terminal. Since the private chat sub-conference is a newly established video conference, the broadcasting rules of the audio and video code streams of each terminal in the video conference are also applicable to the private chat sub-conference, that is, the audio and video code streams of both the first video conferencing terminal and the second video conferencing terminal can be received by the other party.

[0035] S130, sending code streams of the second video conferencing terminal and the third video conferencing terminal to the first video conferencing terminal;

[0036] S140, sending the code stream of the first video conferencing terminal to the second video conferencing terminal through the second multipoint control unit, and also sending the code stream of the third video conferencing terminal to the second video conferencing terminal through the second multipoint control unit.

[0037] In order to enable the first video conferencing terminal and the second video conferencing terminal in the private chat sub-conference to both receive the audio and video code streams of the broadcast source, the first multi-point control unit sends the code stream of the third video conferencing terminal to the private chat sub-conference, so that the first video conferencing terminal and the second video conferencing terminal can simultaneously obtain the picture and sound of the third video conferencing terminal during the private chat. However, since the audio and video data of the first video conferencing terminal and the second video conferencing terminal in the private chat sub-conference are only exchanged between the two, the private chat between the first video conferencing terminal and the second video conferencing terminal will not affect the main venue.

[0038] Through the control method of step S110 to step S140, in the case Figure 1 The establishment of a private chat sub-conference is realized in the cascade video conference shown, which greatly improves the communication effect of the video conference.

[0039] It is worth noting that the code stream of the video conferencing terminal usually includes a video code stream and an audio code stream. Those skilled in the art will know that the video code stream and the audio code stream are exchanged between the video conferencing terminal and the multipoint control unit through different channels. For example, in terms of video code stream transmission, a first video channel and a second video channel are usually established between the video conferencing terminal and the multipoint control unit. The first video channel can be used to transmit the main video code stream, such as camera data, and the second video channel can be used to transmit a video code stream with a slightly smaller amount of data, such as a computer screen. Some video conferencing terminals also support a third video channel or even a fourth video channel for transmitting more video data. The video code stream transmission of the embodiment of the present invention has the following different control methods according to the number of video channels supported by the video conferencing terminal:

[0040] Referring to FIG. 3 , if the first video conferencing terminal supports the third video channel, step S130 specifically includes:

[0041] S132, establishing a first video channel, a second video channel, and a third video channel with the first video conferencing terminal;

[0042] S133, sending the code stream of the third video conferencing terminal to the first video conferencing terminal through the first video channel and / or the second video channel, and sending the video code stream of the second video conferencing terminal to the first video conferencing terminal through the third video channel.

[0043] By using the third video channel to transmit the screen of the private chat sub-conference, it is possible to avoid the video code stream of the private chat sub-conference affecting the video code stream of the third video conferencing terminal, because the main venue (i.e., the third video conferencing terminal) often requires the first video channel and the second video channel to transmit the code stream. In this case, the first video conferencing terminal can connect to three display screens to achieve simultaneous viewing of multiple screens. For example, the first display screen displays the camera screen of the main venue, the second display screen displays the computer screen of the main venue, and the third display screen displays the screen of the second video conferencing terminal. In addition, the first video conferencing terminal can also synthesize the screen of the main venue and the screen of the second video conferencing terminal and play them on one display screen, which can also achieve simultaneous viewing.

[0044] Reference Figure 4 If the first video conferencing terminal does not support the third video channel, step S130 specifically includes:

[0045] S134, establishing a first video channel with the first video conferencing terminal;

[0046] S135, synthesizing the images of the second video conferencing terminal and the third video conferencing terminal, and sending the synthesized image to the first video conferencing terminal through the first video channel.

[0047] Multi-screen synthesis is performed in the first multi-point control unit, and the synthesized video code stream is sent to the first video conferencing terminal through the first video conferencing channel, so as to be played on a display screen of the first video conferencing terminal. Similarly, for the second video conferencing terminal that does not support the third video channel, the second multi-point control unit can also synthesize the images of the first video conferencing terminal and the third video conferencing terminal, and then send them to the second multi-point control unit.

[0048] Before the first multipoint control unit establishes the third video channel with the first video conferencing terminal, the first multipoint control unit needs to first confirm whether the first video conferencing terminal supports the third video channel, which is achieved by the following steps:

[0049] S131, interacting with the first video conferencing terminal about the terminal capability set, and determining that the first video conferencing terminal supports the third video channel capability.

[0050] The above operations are performed by the capability interaction module of the first multipoint control unit. The capability interaction module is used to determine the video format selected for this call according to the capability types of the main video / auxiliary video / private chat video supported by the video conferencing terminal during the video conference call establishment phase. Therefore, it can be understood that after the image acquisition of the third video conferencing terminal is completed, it is compressed by video encoding algorithms such as H.264 and H.265 to become a video code stream and sent to the first multipoint control unit. After receiving the video code stream, the first video conferencing terminal decodes it with a video decoding algorithm that matches the encoding and outputs the image.

[0051] In terms of transmitting audio code streams, in the prior art, the audio data is usually encoded and compressed before being transmitted through an audio channel, and finally decoded and played at the video conferencing terminal. The cascade video conference establishment phase in step S110 of the embodiment of the present invention includes:

[0052] An audio receiving channel and an audio sending channel are established with the first video conferencing terminal. The audio receiving channel is used to receive audio data in the code stream of the first video conferencing terminal. The audio sending channel is used to send audio data in the code stream of the third video conferencing terminal to the first video conferencing terminal.

[0053] Through the audio receiving channel and the audio sending channel, the first video conferencing terminal can receive and send voice, thereby participating in the current cascade video conference.

[0054] After the private chat sub-conference is established, in order to prevent the voice of the first video conferencing terminal from affecting the main venue, the first multipoint control unit creates a new private chat sub-conference for the first video conferencing terminal and the second video conferencing terminal. Since the private chat sub-conference is not completely isolated from the current cascade video conference, the private chat sub-conference also needs to receive the code stream broadcast by the third video conferencing terminal. Therefore, the first multipoint control unit needs to adjust its forwarding rules, which specifically includes the following steps:

[0055] The forwarding destination of the received code stream of the first video conferencing terminal is set to the second video conferencing terminal, and the forwarding destination of the received code stream of the second video conferencing terminal is set to the first video conferencing terminal.

[0056] In the current cascade video conference, the audio channel pre-established by the first multipoint control unit and the first video conferencing terminal is not applicable to the private chat sub-conference, that is, the first multipoint control unit broadcasts the audio code stream of the first video conferencing terminal to all participating terminals. If a private chat sub-conference is established at this time without changing the forwarding rules, the audio code stream of the first video conferencing terminal will affect the main venue. Therefore, the first multipoint control unit adjusts the internal forwarding rules to release related resources so that the audio code streams of the first video conferencing terminal and the second video conferencing terminal are only transmitted between the two, and maintains the sending channel of the audio code stream of the third conference terminal, that is, maintains the channel for sending the audio code stream of the main venue to the first video conferencing terminal and the second video conferencing terminal, so as to realize the function of watching the main venue at the same time during the private chat.

[0057] The process of adjusting the forwarding object further includes: mixing the received voice data in the code stream of the first video conferencing terminal, the voice data in the code stream of the second video conferencing terminal, and the voice data in the code stream of the third video conferencing terminal;

[0058] The mixed voice data is sent to the first video conferencing terminal and the second video conferencing terminal.

[0059] The first multi-point control unit mixes the audio sent by the first video conferencing terminal, the second video conferencing terminal and the third video conferencing terminal, and sends the mixed code stream to the first video conferencing terminal and the second video conferencing terminal. The first video conferencing terminal removes the voice of the first video conferencing terminal from the mixed data based on the Acoustic Echo Cancellation (AEC) function, thereby playing the voices of the second video conferencing terminal and the third video conferencing terminal on the first video conferencing terminal.

[0060] Embodiment 1 of the present invention is a control method for establishing a private chat sub-conference based on a cascaded video conference. The control method can facilitate participating video conference terminals to establish a private chat sub-conference for communication without affecting the progress of the main conference, thereby improving the user experience of the video conference.

[0061] In another embodiment, referring to Figure 5 The cascade composition of the video conferencing system is as follows: it also includes a third multi-point control unit, the first video conferencing terminal (represented by terminal A in the figure) is directly connected to the first multi-point control unit, the second video conferencing terminal (represented by terminal B in the figure) is directly connected to the second multi-point control unit, and the third video conferencing terminal (represented by terminal C in the figure) is directly connected to the third multi-point control unit, wherein the third multi-point control unit as the top-level multi-point control unit is respectively connected to the first multi-point control unit and the second multi-point control unit, the third video conferencing terminal serves as the main venue, and the third multi-point control unit broadcasts the code stream of the third video conferencing terminal to each of the following video conferencing terminals.

[0062] like Figure 6 As shown, Figure 6 is a flow chart of a control method of the video conferencing system in the above embodiment, the control method is applied to the first multipoint control unit, and comprises the following steps:

[0063] S210, establishing a connection with the first video conferencing terminal, and establishing a connection with the third video conferencing terminal through the third multipoint control unit;

[0064] S220, receiving a code stream from the first video conferencing terminal, and receiving a code stream from the third video conferencing terminal through a third multipoint control unit;

[0065] In this embodiment, the first multi-point control unit is connected to the third video conferencing terminal through the third multi-point control unit, thereby receiving the code stream of the third video conferencing terminal serving as the main venue, and forwarding it to the first video conferencing terminal. At the same time, the code stream of the first video conferencing terminal also needs to be broadcast in the current cascade video conference. Therefore, from the perspective of the first multi-point control unit, the first multi-point control unit needs to receive the code stream of the first video conferencing terminal and receive the code stream of the third video conferencing terminal through the third multi-point control unit.

[0066] S230, in response to a private chat conference request initiated by the first video conferencing terminal to the second video conferencing terminal, obtaining a code stream of the second video conferencing terminal through the second multipoint control unit, and the second video conferencing terminal is connected to the second multipoint control unit;

[0067] When the first video conference terminal needs to have a private chat with the second video conference terminal, the first video conference terminal sends a private chat conference request to the first multipoint control unit. After receiving the private chat conference request initiated by the first video conference terminal, the first multipoint control unit establishes a private chat sub-conference for the first video conference terminal and the second video conference terminal. Since the private chat sub-conference is a newly established video conference, the broadcast rules of the audio and video streams of each terminal in the video conference also apply to the private chat sub-conference, that is, the audio and video streams of both the first video conference terminal and the second video conference terminal can be received by the other party. Of course, since the first multipoint control unit and the second multipoint control unit are not directly connected at this time, data can be transmitted through a third multipoint control unit as an intermediary, or a new channel can be established between the first multipoint control unit and the second multipoint control unit for data transmission.

[0068] S240, sending code streams of the second video conferencing terminal and the third video conferencing terminal to the first video conferencing terminal;

[0069] S250: Send the code stream of the first video conferencing terminal to the second video conferencing terminal through the second multipoint control unit.

[0070] The first multipoint control unit obtains the code stream of the third video conferencing terminal through the third multipoint control unit and sends it to the first video conferencing terminal, so that the first video conferencing terminal can simultaneously obtain the picture and sound of the third video conferencing terminal during the private chat. However, since the audio and video data of the first video conferencing terminal and the second video conferencing terminal in the private chat sub-conference are only exchanged between the two, the private chat between the first video conferencing terminal and the second video conferencing terminal will not affect the main venue. Similarly, for the second video conferencing terminal, the second multipoint control unit obtains the code stream of the third video conferencing terminal through the third multipoint control unit and sends it to the second video conferencing terminal, so that the second video conferencing terminal can simultaneously obtain the picture and sound of the third video conferencing terminal during the private chat.

[0071] It can be understood that, similar to Example 1, the code stream of the video conferencing terminal generally includes a video code stream and an audio code stream. The only difference between this embodiment and Example 1 is that the first multi-point control unit is not directly connected to the third video conferencing terminal, but is connected to the third video conferencing terminal through a third multi-point control unit. Therefore, in terms of the transmission of video code streams and audio code streams, this embodiment is the same as the description of Example 1. In order to avoid repetition, it is omitted here and not explained again.

[0072] In another embodiment, referring to Figure 7The video conferencing system includes two independent video conferencing venues, which are composed as follows: the first multi-point control unit is directly connected to the first video conferencing terminal (represented by terminal A in the figure), and the second multi-point control unit is directly connected to the second video conferencing terminal (represented by terminal B in the figure). The first multi-point control unit and the second multi-point control belong to two different venues. For the convenience of explanation, the fourth video conferencing terminal (represented by terminal D in the figure) is introduced as another main venue. In this way, the main venue terminal of the first multi-point control unit is the third video conferencing terminal (represented by terminal C in the figure), and the main venue terminal of the second multi-point control unit is the fourth video conferencing terminal.

[0073] like Figure 8 As shown, Figure 8 is a flow chart of a control method of the video conferencing system in the above embodiment, the control method is applied to the first multipoint control unit, and comprises the following steps:

[0074] S310: Establishing connections with the first video conferencing terminal and the third video conferencing terminal respectively, and receiving code streams from the first video conferencing terminal and the third video conferencing terminal;

[0075] In this embodiment, the first multi-point control unit is directly connected to the third video conferencing terminal and the third video conferencing terminal is selected as the main venue, and the first video conferencing terminal receives the broadcast code stream of the third video conferencing terminal. Since the second multi-point control unit and the first multi-point control unit belong to different venues, the two venues are independent of each other. Therefore, only the operation of the first multi-point control unit is described in this step. It can be understood that for the second multi-point control unit, it is also necessary to directly connect to the fourth video conferencing terminal and select the fourth video conferencing terminal as the main venue, and the second video conferencing terminal receives the broadcast code stream of the fourth video conferencing terminal.

[0076] S320: In response to a private chat conference request initiated by the first video conferencing terminal to the second video conferencing terminal, obtaining a code stream of the second video conferencing terminal through the second multipoint control unit;

[0077] When the first video conferencing terminal needs to have a private chat with the second video conferencing terminal, the first video conferencing terminal sends a private chat conference request to the first multi-point control unit. After receiving the private chat conference request initiated by the first video conferencing terminal, the first multi-point control unit establishes a private chat sub-conference for the first video conferencing terminal and the second video conferencing terminal. At this time, the first multi-point control unit establishes a connection with the second multi-point control unit through a network or other means, and transmits code stream data through a newly established signal; since the private chat sub-conference is a newly established video conference, the broadcast rules of the audio and video code streams of each terminal in the video conference are also applicable to the private chat sub-conference, that is, the audio and video code streams of both the first video conferencing terminal and the second video conferencing terminal can be received by the other party.

[0078] S330, sending the code streams of the second video conferencing terminal and the third video conferencing terminal to the first video conferencing terminal;

[0079] S340: Send the code stream of the first video conferencing terminal to the second video conferencing terminal through the second multipoint control unit.

[0080] Since the first video conferencing terminal and the second video conferencing terminal receive different broadcast sources respectively, from the perspective of the first multi-point control unit, only the received code streams of the second video conferencing terminal and the third video conferencing terminal are sent to the first video conferencing terminal. At the same time, in order to enable the second video conferencing terminal to receive the audio and video data of the first video conferencing terminal, the first multi-point control unit also sends the code stream of the first video conferencing terminal to the second multi-point control unit, and the second multi-point control unit sends the received code streams of the first video conferencing terminal and the fourth video conferencing terminal to the second video conferencing terminal, thereby realizing private chat between the first video conferencing terminal and the second video conferencing terminal.

[0081] Through the above steps, a private chat sub-conference is established for the first video conferencing terminal and the second video conferencing terminal that need to have a private chat. Since the private chat sub-conference is a new video conferencing venue, the audio and video data of the first video conferencing terminal and the second video conferencing terminal are only exchanged between the two. The private chat between the first video conferencing terminal and the second video conferencing terminal will not affect their respective main venues, which greatly improves the communication effect of the video conference.

[0082] It is worth noting that the code stream of the video conferencing terminal usually includes a video code stream and an audio code stream. Those skilled in the art will know that the video code stream and the audio code stream are exchanged between the video conferencing terminal and the multipoint control unit through different channels. For example, in terms of video code stream transmission, a first video channel and a second video channel are usually established between the video conferencing terminal and the multipoint control unit. The first video channel can be used to transmit the main video code stream, such as camera data, and the second video channel can be used to transmit a video code stream with a slightly smaller amount of data, such as a computer screen. Some video conferencing terminals also support a third video channel or even a fourth video channel for transmitting more video data. The video code stream transmission of the embodiment of the present invention has the following different control methods according to the number of video channels supported by the video conferencing terminal:

[0083] Reference Fig. 9 If the first video conferencing terminal supports the third video channel, step S330 specifically includes:

[0084] S332, establishing a first video channel, a second video channel, and a third video channel with the first video conferencing terminal;

[0085] S333, sending the code stream of the third video conferencing terminal to the first video conferencing terminal through the first video channel and / or the second video channel, and sending the video code stream of the second video conferencing terminal to the first video conferencing terminal through the third video channel.

[0086] Using the third video channel to transmit the screen of the private chat sub-conference can avoid the video code stream of the private chat sub-conference from affecting the video code stream of the third video conferencing terminal, because the main venue (i.e., the third video conferencing terminal) often needs the first video channel and the second video channel to transmit the code stream. In this case, the first video conferencing terminal can achieve simultaneous viewing of multiple screens by connecting three display screens. For example, the first display screen displays the camera screen of the main venue, the second display screen displays the computer screen of the main venue, and the third display screen displays the screen of the second video conferencing terminal; in addition, the first video conferencing terminal can also synthesize the screen of the main venue and the screen of the second video conferencing terminal, and play them on one display screen, which can also achieve simultaneous viewing. Similarly, for the second multi-point control unit, a third video channel can also be established with the second video conferencing terminal that supports the third video channel, and the video code stream of the private chat sub-conference can be transmitted through the third video channel.

[0087] Reference Fig.10 If the first video conferencing terminal does not support the third video channel, step S330 specifically includes:

[0088] S334, establishing a first video channel with the first video conferencing terminal;

[0089] S335, synthesize the images of the second video conferencing terminal and the third video conferencing terminal, and send the synthesized image to the first video conferencing terminal through the first video channel.

[0090] Multi-screen synthesis is performed in the first multi-point control unit, and the synthesized video code stream is sent to the first video conferencing terminal through the first video conferencing channel, so as to be played on a display screen of the first video conferencing terminal. Similarly, for the second video conferencing terminal that does not support the third video channel, the second multi-point control unit can also synthesize the images of the first video conferencing terminal and the third video conferencing terminal, and then send them to the second multi-point control unit.

[0091] Before the first multipoint control unit establishes the third video channel with the first video conferencing terminal, the first multipoint control unit needs to first confirm whether the first video conferencing terminal supports the third video channel, which is achieved by the following steps:

[0092] S331, interacting with the first video conferencing terminal about the terminal capability set, and determining that the first video conferencing terminal supports the third video channel capability.

[0093] The above steps are performed by the capability interaction module of the first multipoint control unit. The capability interaction module is used to determine the video format selected for this call according to the capability types of the main video / auxiliary video / private chat video supported by the video conferencing terminal during the video conference call establishment phase. Therefore, it can be understood that after the image acquisition of the third video conferencing terminal is completed, it is compressed by video encoding algorithms such as H.264 and H.265 to become a video code stream and sent to the first multipoint control unit. After receiving the video code stream, the first video conferencing terminal decodes it with a video decoding algorithm that matches the encoding and outputs the image.

[0094] It is understandable that, since the structural composition of the first video conference terminal-the first multipoint control unit-the third video conference terminal is substantially the same as the structural composition of the second video conference terminal-the second multipoint control unit-the fourth video conference terminal, the above control method for the first multipoint control unit is also applicable to the second multipoint control unit. It is worth noting that there may be other multipoint control units between the second multipoint control unit and the second video conference terminal, thereby forming a cascade structure. In this case, the control method for the second multipoint control unit can refer to the steps of Example 1, and will not be described in detail here.

[0095] In terms of transmitting audio code streams, in the prior art, audio data is usually encoded and compressed before being transmitted through an audio channel, and finally decoded and played at the video conferencing terminal. The video conference establishment phase in step S310 of the embodiment of the present invention includes the following steps:

[0096] An audio receiving channel and an audio sending channel are established with the first video conferencing terminal. The audio receiving channel is used to receive audio data in the code stream of the first video conferencing terminal. The audio sending channel is used to send audio data in the code stream of the third video conferencing terminal to the first video conferencing terminal.

[0097] Through the audio receiving channel and the audio sending channel, the first video conferencing terminal can receive and send voice, thereby participating in the current video conference.

[0098] After the private chat sub-conference is established, in order to prevent the voice of the first video conferencing terminal from affecting the main venue, the first multi-point control unit creates a new private chat sub-conference for the first video conferencing terminal and the second video conferencing terminal. Since the private chat sub-conference is not completely isolated from the current video conference, the first multi-point control unit is also connected to the third video conferencing terminal and sends the code stream of the third video conferencing terminal to the first video conferencing terminal. The second multi-point control unit also needs to send the code stream of the fourth video conferencing terminal to the second video conferencing terminal. Therefore, taking the first multi-point control unit as an example, during the processing of the audio code stream, the first multi-point control unit needs to adjust its own forwarding rules, which specifically includes the following steps:

[0099] The forwarding destination of the received code stream of the first video conferencing terminal is set to the second video conferencing terminal, and the forwarding destination of the received code stream of the second video conferencing terminal is set to the first video conferencing terminal.

[0100] In the video conference where the first multipoint control unit is located, the audio channel pre-established by the first multipoint control unit and the first video conferencing terminal is not applicable to the private chat sub-conference, that is, the first multipoint control unit broadcasts the audio code stream of the first video conferencing terminal to all participating terminals. If a private chat sub-conference is established at this time without changing the forwarding rules, the audio code stream of the first video conferencing terminal will affect the main venue. Therefore, the first multipoint control unit adjusts the internal forwarding rules to release relevant resources so that the audio code streams of the first video conferencing terminal and the second video conferencing terminal are only transmitted between the two, and maintains the sending channel of the audio code stream of the third conference terminal, that is, maintains the channel for sending the audio code stream of the main venue to the first video conferencing terminal, so as to realize the function of watching the main venue at the same time during the private chat. Similarly, the second multipoint control unit can also process the audio code stream according to the above steps, the difference is that the audio code stream of the third video conferencing terminal is replaced with the code stream of the fourth video conferencing terminal.

[0101] The process of adjusting the forwarding object further includes: mixing the received voice data in the code stream of the first video conferencing terminal, the voice data in the code stream of the second video conferencing terminal, and the voice data in the code stream of the third video conferencing terminal;

[0102] The mixed voice data is sent to the first video conferencing terminal.

[0103] The first multipoint control unit mixes the audio data sent by the first video conference terminal, the second video conference terminal, and the third video conference terminal, and sends the mixed code stream to the first video conference terminal. The first video conference terminal removes the voice of the first video conference terminal from the mixed data based on the Acoustic Echo Cancellation (AEC) function, so that the voices of the second video conference terminal and the third video conference terminal are played on the first video conference terminal. For the second multipoint control unit, the mixing object includes the audio data sent by the first video conference terminal, the second video conference terminal, and the fourth video conference terminal. The second video conference terminal plays the voices of the first video conference terminal and the fourth video conference terminal after filtering through the AEC function.

[0104] The control method of an embodiment of the present invention is that, in the current multi-level video conference, when the first multi-point control unit receives a private chat conference request initiated by the first video conferencing terminal to the second video conferencing terminal, the first multi-point control unit creates a new sub-conference for the first video conferencing terminal and the second video conferencing terminal, and controls the exchange of audio and video data between the first video conferencing terminal and the second video conferencing terminal, and at the same time provides the first video conferencing terminal with audio and video data of the third video conferencing terminal as a broadcast source. In this way, the first video conferencing terminal and the second video conferencing terminal can establish a private chat sub-conference in the multi-level video conference, and the audio and video data of the private chat will not affect the third video conferencing terminal as a broadcast source, which brings convenience to the application of video conferencing.

[0105] Another embodiment of the present invention further provides a multipoint control unit 400, referring to Fig.11 The multipoint control unit 400 includes a memory 402, a processor 401, and a computer program stored in the memory 402 and executable on the processor 401. When the computer program is executed by the processor 401, any of the aforementioned video conferencing system control methods is implemented. The multipoint control unit 400 uses the control method in the embodiment of the present invention to establish a sub-conference for the first video conferencing terminal and the second video conferencing terminal through the control method in the embodiment of the present invention. The multipoint control unit 400 controls the exchange of audio and video data between the first video conferencing terminal and the second video conferencing terminal, and provides the first video conferencing terminal with audio and video data of the third video conferencing terminal as a broadcast source. Thus, the first video conferencing terminal and the second video conferencing terminal establish a sub-conference for private chat under the multi-level video conferencing, and the audio and video data of the private chat will not affect the third video conferencing terminal as a broadcast source, which brings convenience to the application of video conferencing.

[0106] Another embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor 401 or a controller, for example, by a processor 401 in the multipoint control unit 400 embodiment, so that the processor 401 can execute the control method in the above embodiment, for example, execute the above described Figure 2 Steps S110 to S140 of the method, Figure 3 Steps S131 to S133 of the method, Figure 4 Steps S134 to S135 of the method, Figure 6 Steps S210 to S250 of the method, Figure 8 Steps S310 to S340 of the method, Fig. 9Steps S331 to S333 of the method, Fig.10 Method steps S334 to S335.

[0107] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0108] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A control method for a video conferencing system, the video conferencing system comprising a first video conferencing terminal, a second video conferencing terminal, a third video conferencing terminal, a first multipoint control unit and a second multipoint control unit, the control method being applied to the first multipoint control unit, the control method comprising: Establishing connections with the first video conferencing terminal and the third video conferencing terminal respectively, and receiving code streams from the first video conferencing terminal and the third video conferencing terminal; In response to a private chat conference request initiated by the first video conferencing terminal to the second video conferencing terminal, acquiring a code stream of the second video conferencing terminal through the second multipoint control unit, wherein the second video conferencing terminal is connected to the second multipoint control unit; Sending code streams of the second video conferencing terminal and the third video conferencing terminal to the first video conferencing terminal; Sending the code stream of the first video conferencing terminal to the second video conferencing terminal through the second multipoint control unit; The sending of the code streams of the second video conferencing terminal and the third video conferencing terminal to the first video conferencing terminal comprises: Establishing a first video channel, a second video channel, and a third video channel with the first video conferencing terminal; The code stream of the third video conferencing terminal is sent to the first video conferencing terminal through the first video channel and / or the second video channel, and the video code stream of the second video conferencing terminal is sent to the first video conferencing terminal through the third video channel.

2. The control method of the video conferencing system according to claim 1, characterized in that: When the code stream of the first video conferencing terminal is sent to the second video conferencing terminal through the second multipoint control unit, the code stream of the third video conferencing terminal is also sent to the second video conferencing terminal through the second multipoint control unit.

3. The control method of the video conferencing system according to claim 1, characterized in that: The video conferencing system further includes a third multipoint control unit; The establishing of connections with the first video conferencing terminal and the third video conferencing terminal respectively, and receiving code streams from the first video conferencing terminal and the third video conferencing terminal, comprises: Establishing a connection with the first video conferencing terminal, and establishing a connection with the third video conferencing terminal through the third multipoint control unit; Receive the code stream from the first video conferencing terminal, and receive the code stream from the third video conferencing terminal through the third multipoint control unit.

4. The control method of the video conferencing system according to claim 1, characterized in that: The sending of the code streams of the second video conferencing terminal and the third video conferencing terminal to the first video conferencing terminal comprises: Establishing a first video channel with the first video conferencing terminal; The images of the second video conferencing terminal and the third video conferencing terminal are synthesized, and the synthesized images are sent to the first video conferencing terminal through the first video channel.

5. The control method of the video conferencing system according to claim 1, characterized in that: The establishing connections with the first video conferencing terminal and the third video conferencing terminal respectively includes: An audio receiving channel and an audio sending channel are established with the first video conferencing terminal, the audio receiving channel is used to receive the audio data in the code stream of the first video conferencing terminal, and the audio sending channel is used to send the audio data in the code stream of the third video conferencing terminal to the first video conferencing terminal.

6. The control method of the video conferencing system according to claim 1, characterized in that: When the first multipoint control unit establishes a connection with the second multipoint control unit, the control method further includes: Setting the forwarding destination of the received code stream of the first video conferencing terminal to the second video conferencing terminal; And, the forwarding destination of the received code stream of the second video conferencing terminal is set to the first video conferencing terminal.

7. The control method of the video conference system according to claim 6, characterized in that: When the first multipoint control unit establishes a connection with the second multipoint control unit, the control method further includes: The received voice data in the code stream of the first video conferencing terminal, the voice data in the code stream of the second video conferencing terminal, and the voice data in the code stream of the third video conferencing terminal are mixed.

8. A multipoint control unit, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the control method of the video conferencing system according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an information processing program, and when the information processing program is executed by the processor, the control method of the video conferencing system according to any one of claims 1 to 7 is implemented.

Images