Audio and video stream scheduling method, device, command and scheduling terminal and storage medium

Abstract

An embodiment of the present invention provides an audio and video stream scheduling method, an apparatus, a command and dispatch terminal and a storage medium. In the embodiment of the present invention, the command and dispatch terminal receives a playback signal sent by a Web server for playing the audio and video stream of a first participant terminal; and based on the playback signal, determines whether the playback permission of a pre-stored second participant terminal includes the audio and video stream of the first participant terminal; when the playback permission of the second participant terminal includes the audio and video stream of the first participant terminal, the command and dispatch terminal generates a flow adjustment request for scheduling the audio and video stream of the first participant terminal to the second participant terminal, and sends the flow adjustment request to the conference scheduling server for flow adjustment, so that the audio and video stream can be transmitted in a targeted manner in a real-time video conferencing scenario, meeting the transmission requirements of audio and video streams between different participant terminals in a video conferencing scenario.

Description

Technical Field

[0001] The present invention relates to the field of visual networking technology, and in particular to an audio and video stream scheduling method, an audio and video stream scheduling device, a command and scheduling terminal, and a storage medium. Background Art

[0002] With the rapid development of network technology, two-way communications such as video conferencing and video teaching have become widely popular in users' lives, work, and study. Video conferencing refers to a meeting in which people in two or more locations have face-to-face conversations through communication equipment and the Internet.

[0003] Video conferencing technology and video conferencing management systems have been widely used in practice and have played a very important role in various video conferences. However, the current conference management system still has some shortcomings. In the current video conferencing, all participating terminals are at the same level, and the audio and video streams that each participating terminal can see are the same, which makes it impossible to conduct some private conference reports in real-time video conferences. For example, in a company video conference, a middle-level leader wants to report to the upper-level leader, but does not want his subordinates to see it. The existing video conferencing cannot meet the needs of this scenario. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide an audio and video stream scheduling method, an audio and video stream scheduling device, a command and scheduling terminal, and a storage medium that overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the present invention, an embodiment of the present invention discloses an audio and video stream scheduling method, which is applied to a command and dispatch end. The method includes:

[0006] Receiving a play signaling sent by the Web server for playing the audio and video stream of the first conference participant terminal;

[0007] For the play signaling, determining whether the pre-stored play permission of the second participant terminal includes the audio and video stream of the first participant terminal;

[0008] When the playback permission of the second participant terminal includes the audio and video stream of the first participant terminal, generating a stream modulation request for scheduling the audio and video stream of the first participant terminal to the second participant terminal;

[0009] The flow adjustment request is sent to the conference scheduling server for flow adjustment.

[0010] Optionally, the first conference participant terminal and the second conference participant terminal both have a conference participant level, and the conference participant levels of the plurality of second conference participant terminals and the first conference participant terminal are in a tree structure;

[0011] The playback authority of the second participating terminal is determined according to the participating level of the second participating terminal in the tree structure.

[0012] Optionally, for the play signaling, determining whether the pre-stored play permission of the second participant terminal includes the audio and video stream of the first participant terminal includes:

[0013] For the play signaling, determining a relationship between a participation level of the second participating terminal and a participation level of the first participating terminal in the tree structure;

[0014] According to the relationship between the participation level of the second participating terminal and the participation level of the first participating terminal in the tree structure, it is determined whether the pre-stored playback permission of the second participating terminal includes the audio and video stream of the first participating terminal.

[0015] Optionally, the first conference participant terminal has a first terminal number, and the second conference participant terminal has a second terminal number;

[0016] When the playback permission of the second conference participant terminal includes the audio and video stream of the first conference participant terminal, generating a stream modulation request for scheduling the audio and video stream of the first conference participant terminal to the second conference participant terminal includes:

[0017] When the playback permission of the second participant terminal includes the audio and video stream of the first participant terminal, obtaining a first terminal number of the first participant terminal and a second terminal number of the second participant terminal;

[0018] The first terminal number and the second terminal number are encapsulated as the stream modulation request to dispatch the audio and video stream of the first participant terminal to the second participant terminal.

[0019] Optionally, the play signaling further includes a third terminal number of a third participant terminal for specifying the play of the audio and video stream of the first participant terminal; the method further includes:

[0020] For the third terminal number, determining whether the playback permission of the third participating terminal is pre-stored;

[0021] When the playback permission of the third participant terminal is pre-stored, it is determined whether the playback permission of the third participant terminal includes the audio and video stream of the first participant terminal.

[0022] Optionally, the method further includes:

[0023] When the playback permission of the third conference participant terminal is not pre-stored, generating an error message;

[0024] The error message is sent to the Web server.

[0025] According to another aspect of the present invention, an embodiment of the present invention further discloses a command and dispatch terminal, including a dispatch module, a conference module, a feedback module, a level data module and a conference dispatch protocol proxy module, wherein:

[0026] The level data module is connected to the scheduling module and is used to store the playback authority of the second participant terminal;

[0027] The scheduling module is connected to the Web server, the conference module and the conference scheduling protocol proxy module, and is used to send the play signaling sent by the Web server for playing the audio and video stream of the first conference participant terminal to the conference module and the conference scheduling protocol proxy module, and is used to send the play permission of the second conference participant terminal in the level data module to the conference module;

[0028] The conference module is connected to the feedback module, and is used to determine whether the playback permission of the second participant terminal includes the audio and video stream of the first participant terminal for the playback signaling, and send the judgment result to the feedback module;

[0029] The feedback module is connected to the level data module and the conference scheduling protocol proxy module, and is used to send the judgment result to the conference scheduling protocol proxy module when the judgment result is that the playback permission of the second conference participant terminal includes the audio and video stream of the first conference participant terminal;

[0030] The conference scheduling protocol proxy module is connected to the conference scheduling server and the level data module, and is used to generate a flow adjustment request for scheduling the audio and video stream of the first participant terminal to the second participant terminal according to the judgment result sent by the feedback module, and send the flow adjustment request to the conference scheduling server for flow adjustment.

[0031] According to another aspect of the present invention, an embodiment of the present invention further discloses an audio and video stream scheduling device, which is applied to a command and scheduling end, and the device includes:

[0032] A play signaling receiving module, used to receive a play signaling sent by a Web server for playing the audio and video stream of the first conference participant terminal;

[0033] A first playback permission judgment module, used for judging, with respect to the playback signaling, whether the playback permission of the second conference participant terminal stored in advance includes the audio and video stream of the first conference participant terminal;

[0034] a stream adjustment request generation module, configured to generate a stream adjustment request for scheduling the audio and video stream of the first conference participant terminal to the second conference participant terminal when the playback permission of the second conference participant terminal includes the audio and video stream of the first conference participant terminal;

[0035] The flow adjustment request sending module is used to send the flow adjustment request to the conference scheduling server for flow adjustment.

[0036] According to another aspect of the present invention, an embodiment of the present invention further discloses an audio and video stream scheduling device, comprising:

[0037] one or more processors; and

[0038] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the device to perform the above-mentioned audio and video stream scheduling method.

[0039] According to another aspect of the present invention, an embodiment of the present invention further discloses a computer-readable storage medium, wherein a computer program stored in the storage medium enables a processor to execute the above-mentioned audio and video stream scheduling method.

[0040] The embodiments of the present invention include the following advantages:

[0041] In an embodiment of the present invention, the playback permissions of the second participant terminal are pre-stored in the command and dispatch end. When the command and dispatch end receives the playback signaling sent by the Web background service for playing the audio and video stream of the first participant terminal, not all second participant terminals have the permission to play the audio and video stream of the first participant terminal. The command and dispatch end will perform a screening based on the playback permissions of each second participant terminal, and then dispatch the audio and video stream of the first participant terminal to the second participant terminal with the playback permission of the audio and video stream of the first participant terminal. In this way, the audio and video stream can be transmitted in a targeted manner in a real-time video conferencing scenario, meeting the audio and video stream transmission requirements between different participant terminals in a video conferencing scenario.

[0042] In an embodiment of the present invention, the participating terminals in the video conference also have a participating level, and the participating level of each participating terminal is a tree structure. The embodiment of the present invention also proposes a method for determining the playback permission of the second participating terminal based on the participating level of the second participating terminal. The playback permission of the second participating terminal can be screened according to the relationship between the participating level of the second participating terminal and the participating level of the first participating terminal in the tree structure. In this way, different audio and video streams can be scheduled between participating terminals of different participating levels, which solves the problem of transmitting audio and video streams in the network under the hierarchical structure and improves the privacy and security in the video conferencing scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1This is a flowchart of the steps of Embodiment 1 of an audio and video stream scheduling method of the present invention;

[0044] Figure 2 It is a schematic diagram of the network between the command and dispatch terminal and the Web background service and the CMS meeting management service in an embodiment of the present invention;

[0045] Figure 3 This is a flowchart of the steps of Embodiment 2 of an audio and video stream scheduling method of the present invention;

[0046] Figure 4 is a schematic diagram of a hierarchical structure of an optional example of embodiment 1 of the present invention;

[0047] Figure 5 It is a structural diagram of a command and dispatch terminal according to an embodiment of the present invention;

[0048] Figure 6 It is a structural diagram of a third embodiment of an audio and video stream scheduling device of the present invention. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Reference Figure 1 , shows a flowchart of the steps of Embodiment 1 of an audio and video stream scheduling method of the present invention, which can be applied to a command and dispatch end. The method may specifically include the following steps:

[0051] Step S101, receiving a play signaling sent by a Web server for playing an audio or video stream of a first conference participant terminal;

[0052] In various embodiments of the present invention, the Web server may be a server with a Web background service installed, and the Web background service may be based on Web Service technology, which enables different applications running on different machines to exchange data or integrate with each other without the aid of additional, specialized third-party software or hardware.

[0053] The conference scheduling server can be a server installed with a visual network conference scheduling system. The visual network conference scheduling system (also known as a CMS conference management server) can exchange data with the command and dispatch end based on a 64-bit visual network conference management protocol.

[0054] The command and dispatch end can also be called the command and dispatch middle layer. It is a service software that can be deployed on a server with the Web background service, or deployed on a server with the CMS management service, or deployed on a single machine. Figure 2, showing the connection relationship between the command and dispatch middle layer and the Web background service and the CMS meeting management service in an embodiment of the present invention.

[0055] In a video conference, the Web background service can provide a conference scheduling interface for the conference host. At the beginning of the meeting, the Web server pulls in each participating terminal to form a list of conference members, which can be presented in the form of a list in the conference scheduling interface. There can also be a video display window on the conference scheduling interface, and the video display window can be displayed in a single-screen or multi-screen manner, that is, it can be displayed as a single video screen or as multiple video screens. Each video screen displays the audio and video stream being played in the current video conference. This audio and video stream can be the audio and video screen of the participating terminal that is currently in the speaking state, or it can be the audio and video screen of the participating terminal that is currently in the watching state. The participating terminal mentioned in each embodiment of the present invention refers to the device that ultimately encodes and decodes the audio and video stream. The commonly used terminals in the visual network include Aurora, Qiming, Aurora 4K, etc.

[0056] Based on the above connection relationship, the command and dispatch end of the embodiment of the present invention can receive the play signaling sent by the Web background service in the Web server for playing the audio and video stream of the first participant terminal. The play signaling can be generated by the Web background service according to the operation of the conference host dragging the first participant terminal in the conference member list into the video screen by dragging or selecting and clicking in the conference scheduling interface, and then the Web server sends the play signaling to the command and dispatch end. In implementation, since the video screen in the video display window can be one or more, the first participant terminal selected by the conference host can be one or more, that is, the play signaling of the embodiment of the present invention indicates that the audio and video stream to be played can be the audio and video stream of one first participant terminal or the audio and video stream of multiple first participant terminals.

[0057] Step S102: for the play signaling, determining whether the pre-stored play permission of the second participant terminal includes the audio and video stream of the first participant terminal;

[0058] In an embodiment of the present invention, the command and dispatch end may pre-store the playback permissions of one or more second participant terminals. The playback permissions of the second participant terminals may be sent to the command and dispatch end by the Web background service when the conference is started. The second participant terminal referred to in the embodiment of the present invention may refer to some or all of the participant terminals in the conference member list, so that the playback permissions of specific participant terminals in a video conference can be managed to suit different application scenarios. It should be noted that the first participant terminal and the second participant terminal in the embodiment of the present invention are only used to distinguish the description. In practice, the second participant terminal may also refer to the first participant terminal, that is, the playback permissions of the first participant terminal may include its own audio and video streams.

[0059] In an optional embodiment of the present invention, the first participating terminal and the second participating terminal both have a participating level, and the participating levels of multiple second participating terminals and the first participating terminal are in a tree structure; wherein the playback permission of the second participating terminal is determined based on the participating level of the second participating terminal in the tree structure.

[0060] The conference level in the embodiment of the present invention may refer to the position level in the company, or the jurisdiction level in the government, or other levels. The conference levels of multiple conference terminals are in a tree structure. The playback authority of the second conference terminal can be determined in the following ways:

[0061] For example, in a company video conference, a middle-level leader needs to report to the upper-level leader, but does not want his subordinates to see it. Then the conference terminal used by the middle-level leader is the first conference terminal, and the audio and video stream of the first conference terminal is the audio and video picture of the middle-level leader. Both the upper-level leader and his subordinates can be the second conference terminal. Then it can be predetermined that the upper-level leader's playback authority can include the audio and video stream of his direct subordinates (middle-level leaders), while the subordinates' playback authority does not include the audio and video stream of their direct superiors (middle-level leaders).

[0062] For example, a middle-level leader wants to assign work to his subordinates but does not want the upper-level leader to see it. In this case, the subordinate’s playback permission may include the audio and video streams of his immediate superior (middle-level leader), while the upper-level leader’s playback permission does not include the audio and video streams of his immediate subordinate (middle-level leader).

[0063] For example, in a provincial government meeting, the playback rights of the second participant terminal are pre-set to include only the audio and video streams of its immediate superior and the highest superior. Then the playback rights of the county-level participant terminal (such as the county magistrate) can include the audio and video streams of the city-level participant terminal (such as the mayor) of the city under its jurisdiction and the audio and video streams of the provincial-level participant terminal (such as the governor). In particular, still taking the provincial government meeting as an example, the second participant terminal is the governor, and its playback rights can include the audio and video streams of all the participant terminals in the video conference.

[0064] Based on the participation level of the second participating terminal in the tree structure, the playback permissions of the second participating terminal can be stored in different forms in the command and dispatch terminal, such as a playback permission list that records the relevant identifiers of playable audio and video streams, or a hierarchical relationship list that records the relationship between the participation level of the second participating terminal and the participation level of the participating terminals with playback permissions in the tree structure.

[0065] The first is that when the storage form of the playback permission of the second participant terminal is a playback permission list, the playable audio and video streams can be distinguished by relevant identifiers on the playback permission list, such as the relevant identifier can refer to the terminal number of the first participant terminal. In this optional embodiment, after receiving the playback signaling, the command and dispatch terminal parses and obtains the relevant identifier (such as the terminal number) of the audio and video stream of the first participant terminal, and then determines whether the pre-stored playback permission of the second participant terminal includes the audio and video stream of the first participant terminal, that is, checks in turn whether the playback permission list of each second participant terminal records the relevant identifier of the audio and video stream of the first participant terminal; if so, it means that the second participant terminal has the playback permission of the audio and video stream of the first participant terminal, otherwise it does not. Since the playback signaling indicates that the audio and video stream to be played can be the audio and video streams of multiple first participant terminals, the playback permission of the second participant terminal may also only include some of the audio and video streams of the first participant terminal. When judging, it is necessary to compare the playback permission list of the second participant terminal with the identifier of the audio and video stream of each first participant terminal one by one. For example, in a company video conference, based on the participation level of the second participating terminal in the tree structure, the playback permissions of the second participating terminal include the audio and video streams of its direct subordinates and the audio and video streams of its direct superiors. There is one direct superior, with terminal number A; there are five direct subordinates, with terminal numbers B1 to B5. Then the playback permission list records the six terminal numbers A, B1 to B5, indicating that the second participating terminal can play the audio and video streams of the participating terminals corresponding to these six terminal numbers. Of course, the playback permission list of the second participating terminal can also record the terminal numbers of other participating terminals, such as the terminal number of the participating terminal used by finance. This way of setting playback permissions is more intuitive and can help to quickly determine whether there is playback permission for the audio and video streams of the first participating terminal.

[0066] The second type is that when the playback permission of the second participant terminal is a preset hierarchical relationship list, the relationship recorded in the hierarchical relationship list can be: direct superior, direct subordinate and highest superior, indicating that the playback permission of the second participant terminal can include the audio and video streams of its direct superior, direct subordinate and highest superior. This setting method can be conducive to the management and maintenance of the playback permission of the second participant terminal. For example, when the participation level of each participant terminal changes due to personnel transfer, the playback permission of the second participant terminal may not be adjusted.

[0067] Based on the second playing permission setting method, step S102 may include the following sub-steps:

[0068] Sub-step 1021, determining, with respect to the play signaling, a relationship between a conference level of the second conference participating terminal and a conference level of the first conference participating terminal in the tree structure;

[0069] Sub-step 1022: determining whether the pre-stored playback permission of the second participating terminal includes the audio and video stream of the first participating terminal according to the relationship between the participating level of the second participating terminal and the participating level of the first participating terminal in the tree structure.

[0070] As for sub-steps 1021 to 1022, for example, the playback authority of the second participant terminal includes the audio and video streams of its immediate superior, immediate subordinate, and highest superior. Then, by judging whether the relationship between the participant level of the second participant terminal and the participant level of the first participant terminal in the tree structure belongs to any one of the three types of immediate superior, immediate subordinate, and highest superior, it can be determined whether the participant terminal has the playback authority of the audio and video stream of the first participant terminal. In an embodiment of the present invention, the relationship between the participant level of the second participant terminal and the participant level of the first participant terminal in the tree structure can be determined based on the corresponding terminal number or other level identifier. When determined by the terminal number, the participant level of each participant terminal can be distinguished by the number of digits of the terminal number or the characters on different digits, thereby determining the relationship in the tree structure. In a feasible example of the first embodiment of the present invention: the terminal number of the second conference participant terminal is A2025, the terminal number of its direct superior is A1000, the terminal number of the highest superior is A0000, A1 is the mark of the direct superior, and A0 is the mark of the highest superior.

[0071] Step S103: when the playback permission of the second participant terminal includes the audio and video stream of the first participant terminal, generating a stream modulation request for scheduling the audio and video stream of the first participant terminal to the second participant terminal;

[0072] The first conference participant terminal has a first terminal number, and the second conference participant terminal has a second terminal number;

[0073] Step S103 may include the following sub-steps:

[0074] Sub-step 1031, when the playback permission of the second participating terminal includes the audio and video stream of the first participating terminal, obtaining a first terminal number of the first participating terminal and a second terminal number of the second participating terminal;

[0075] Sub-step 1032: encapsulate the first terminal number and the second terminal number into the stream modulation request to dispatch the audio and video stream of the first participant terminal to the second participant terminal.

[0076] In an embodiment of the present invention, the first terminal number of the first participating terminal can be obtained from the playback signaling, and the second terminal number of the second participating terminal can be obtained from the pre-stored playback permission list or the hierarchical relationship list, and then the first terminal number and the second terminal number are encapsulated as the stream adjustment request to schedule the audio and video stream of the first participating terminal to the second participating terminal. The encapsulation method of the data packet of the stream adjustment request can refer to the existing technology and will not be described in detail here.

[0077] In specific implementation, since the second participant terminal with the playback permission of the audio and video stream of the first participant terminal can be one or more, and the playback signaling indicates that the audio and video stream to be played can also be more than one. Therefore, the command and dispatch end generates a flow adjustment request for dispatching the audio and video stream of the first participant terminal to the second participant terminal, which can be generated in the following ways: (1) Generate a flow adjustment request for dispatching the audio and video stream of a first participant terminal to multiple second participant terminals with their playback permission; (2) Generate a flow adjustment request for dispatching the audio and video stream of multiple first participant terminals to a second participant terminal with its playback permission; (3) Generate a flow adjustment request for dispatching the audio and video stream of a first participant terminal to a second participant terminal with its playback permission. The embodiment of the present invention adopts (1) (2) encapsulation methods to reduce the number of data packets of the flow adjustment request, and when sending the flow adjustment request to the conference scheduling server, it can reduce the number of data packets sent, which is beneficial to reduce the operating pressure of the server.

[0078] Step S104: sending the flow adjustment request to the conference scheduling server for flow adjustment.

[0079] In an embodiment of the present invention, the command and dispatch end may send a flow adjustment request to the conference scheduling server based on the video network conference management protocol. After receiving the flow adjustment request, the conference scheduling server executes the operation of dispatching the audio and video stream of the first participant terminal to the second participant terminal that has the playback permission of the audio and video stream of the first participant terminal. The above-mentioned flow adjustment process may adopt existing technology, which is not the focus of the present invention, and will not be elaborated here, nor will it be limited.

[0080] After the stream adjustment is completed, only the second participant terminal with the playback permission of the audio and video stream of the first participant terminal can play the audio and video stream of the first participant terminal, which meets the hierarchical transmission requirements in the video conference, and can schedule different audio and video streams between participant terminals of different levels, improving the privacy and security of the video conference scene, and making it more flexible. When a second participant terminal does not have the playback permission of the audio and video stream of the first participant terminal, the second participant terminal will still continue to play the audio and video stream with playback permission received before to ensure the normal conduct of the video conference.

[0081] In order to facilitate those skilled in the art to have a deeper understanding of the present invention, the present invention is further described in detail based on the first embodiment. Figure 3 , shows a flowchart of the steps of Embodiment 2 of an audio and video stream scheduling method of the present invention, which can be applied to a command and dispatch terminal, wherein the first conference participant terminal has a first terminal number, and the method can specifically include the following steps:

[0082] Step S301, receiving a play signaling sent by a Web server for playing an audio and video stream of a first participant terminal, wherein the play signaling includes a first terminal number of the first participant terminal and a third terminal number of a third participant terminal for specifying the audio and video stream of the first participant terminal to be played;

[0083] In an embodiment of the present invention, according to specific needs, the audio and video stream of the first participant terminal is only allowed to be viewed by a small number of participants. Therefore, when the conference host selects the audio and video stream to be played as the audio and video stream of the first participant terminal in the conference scheduling interface, he can also select a third participant terminal in the conference member list to specify the playback of the audio and video stream of the first participant terminal. There can also be multiple third participant terminals.

[0084] In an optional example of the first embodiment of the present invention: the conference host selects the participant terminal with terminal number 1006 to play the audio and video stream of the participant terminal with terminal number 1009, then the terminal number 1009 is the first terminal number, and the participant terminal corresponding to it is the first participant terminal; the terminal number 1006 is the third terminal number, and the participant terminal corresponding to it is the third participant terminal. The playback signaling indicates that the audio and video stream to be played includes the terminal number 1009, indicating that the terminal number used to specify the playback of 1009 also includes 1006.

[0085] Step S302, for the third terminal number, determining whether the playback permission of the third conference participant terminal is pre-stored;

[0086] Continuing with the above example, after the command and dispatch end receives the play signaling, it parses and obtains the terminal number 1006, and first searches to see whether the play permission of the third participating terminal corresponding to the terminal number 1006 is pre-stored.

[0087] When the playback permission of the third conference participant terminal is not pre-stored, an error message is generated; and the error message is sent to the Web server.

[0088] In the embodiment of the present invention, the above error message includes the terminal number 1006. After being sent to the Web server, the Web server can present the error message to the conference host, indicating that the playback permission of the participating terminal with the terminal number 1006 is not pre-stored, and there is an abnormal problem, which is convenient for the conference host to handle in time. This solution is generally applicable to the situation where a new conference member is temporarily added to the video conference, but the playback permission of the conference member is not pre-stored, so as to screen the playback permission.

[0089] Step S303: when the playback permission of the third participant terminal is pre-stored, determining whether the playback permission of the third participant terminal includes the audio and video stream of the first participant terminal;

[0090] As described in step S102, there may be multiple ways of storing the playback permissions of the participating terminals. For the first method, after the command and dispatch end determines that the playback permissions of the third participating terminal are pre-stored, it directly extracts the playback permissions of the third participating terminal, and then queries whether the first terminal number of the first participating terminal is recorded in the playback permission list of the third participating terminal, so as to determine whether there is playback permission for the audio and video stream of the first participating terminal.

[0091] For the first method, in an optional example of an embodiment of the present invention: the playback permission list of the third participant terminal with terminal number 1006 records the terminal numbers 1001 to 1008, and the first terminal number of the first participant terminal is 1009, which indicates that the third participant terminal does not have the playback permission to play the audio and video streams of the first participant terminal. At this time, relevant error information can also be generated and fed back to the Web server. Based on the relevant error information, the conference host can retrieve the playback permission list of the third participant terminal with terminal number 1006 from the command and dispatch end through the Web background service, and then add the terminal number 1009 to the playback permission list of the third participant terminal, and then return the playback permission list after adding 1009 to the command and dispatch end. In this way, the third participant terminal with terminal number 1006 has the permission to play the audio and video streams of the first participant terminal with terminal number 1009.

[0092] For the second method, step S303 may include the following sub-steps:

[0093] Sub-step 3031, determining the relationship between the participation level of the third participating terminal and the participation level of the first participating terminal in the tree structure according to the first terminal number and the third terminal number in the play signaling;

[0094] Sub-step 3032: determining whether the playback permission of the third participating terminal includes the audio and video stream of the first participating terminal according to the relationship between the participating level of the third participating terminal and the participating level of the first participating terminal in the tree structure.

[0095] In an embodiment of the present invention, when the command and dispatch end determines that the playback permission of the third participant terminal is pre-stored, it extracts the hierarchical relationship list of the third participant terminal, and then determines the relationship between the participation level of the third participant terminal and the participation level of the first participant terminal in the tree structure, and then queries whether the hierarchical relationship is recorded in the hierarchical relationship list, thereby determining whether the playback permission of the third participant terminal includes the audio and video streams of the first participant terminal.

[0096] For sub-steps 3031 to 3032, in an optional example of the first embodiment of the present invention: in a government video conference, the participants include the governor, city a, city b, county a1, county a2, county a3 under city a, and county b1, county b2, county b3 under city b; the terminal number of the participating terminal used by the governor is 00020, the terminal number corresponding to city a is 00201, the terminal number corresponding to city b is 00202, the terminal numbers corresponding to county a1, county a2, county a3 are 02011, 02012, 02013 respectively, and the terminal numbers corresponding to county b1, county b2, county b3 are 02025, 02026, 02027 respectively. Figure 4 , showing a schematic diagram of the hierarchical structure of an optional example of embodiment 1 of the present invention.

[0097] According to the meeting requirements, the conference host selects the audio and video streams of the governor, city a, and county b1 as the audio and video streams to be played (i.e., the conference terminals corresponding to the governor, city a, and county b1 refer to the first conference terminal in the embodiment of the present invention), and selects the conference terminals corresponding to county a1, county b1, and county b2 as the third conference terminal for specifying the audio and video streams of the governor, city a, and county b1. Therefore, in the generated playback signaling, the first terminal number includes 00020, 00201, and 02025, and the third terminal number includes 02011, 02025, and 02026.

[0098] Assuming that the command and dispatch end has pre-stored the playback permissions of the third participating terminals with terminal numbers 02011, 02025, and 02026, the command and dispatch end will extract the hierarchical relationship lists of these three third participating terminals respectively. Assuming that their hierarchical relationship lists record the direct superiors, direct subordinates, and the highest superiors, by comparing the third terminal numbers 02011, 02025, and 02026 with the first terminal numbers 02011, 02025, and 02026 in turn, the audio and video streams that can actually be played in a1 county, a2 county, and b2 can be obtained. For example, by comparing 00020, 00201, 02025 and 02011, it is found that the first participant terminal (provincial governor) corresponding to 00020 is the highest superior of the third participant terminal (county a1) corresponding to 02011, the first participant terminal (city a) corresponding to 00201 is the direct superior of county a1, and the first participant terminal (county b1) corresponding to 02025 is the remote peer of county a1. Therefore, in this stream adjustment, county a1 can actually play the audio and video streams of the governor and city a; correspondingly, county b1 can play the audio and video streams of the governor, and county b2 can only play the audio and video streams of the governor.

[0099] Step S304: When the playback permission of the third participant terminal includes the audio and video stream of the first participant terminal, encapsulate the third terminal number and the first terminal number to generate a stream modulation request for scheduling the audio and video stream of the first participant terminal to the third participant terminal;

[0100] Step S305: Send the flow adjustment request to the conference scheduling server for flow adjustment.

[0101] In an embodiment of the present invention, for example, if the third participant terminal with terminal number 1006 has the playback permission for the audio and video stream of the first participant terminal with terminal number 1009, the command and dispatch end will encapsulate 1006 and 1009 together, generate a flow adjustment request for scheduling the audio and video stream of 1009 to the third participant terminal corresponding to 1006, and then send the flow adjustment request to the conference scheduling server. Finally, the conference scheduling server executes the operation of scheduling the audio and video stream of 1009 to the third participant terminal corresponding to 1006.

[0102] Combining steps S301 to S305, the embodiment of the present invention adds a third participant terminal for specifying the playback of the audio and video stream of the first participant terminal on the basis of embodiment one, and proposes a scheduling method for audio and video streams that is more private and targeted in meetings, which can meet the scheduling business needs of participant terminals of different levels in real-time video conferences.

[0103] Reference Figure 5, based on the same inventive concept as the first embodiment, shows a schematic diagram of the structure of a command and dispatch terminal according to an embodiment of the present invention, which may include a dispatch module, a conference module, a feedback module, a level data module, and a conference scheduling protocol proxy module, wherein:

[0104] The level data module is connected to the scheduling module and is used to store the playback authority of the second participant terminal;

[0105] The scheduling module is connected to the Web server, the conference module and the conference scheduling protocol proxy module, and is used to send the play signaling sent by the Web server for playing the audio and video stream of the first conference participant terminal to the conference module and the conference scheduling protocol proxy module, and is used to send the play permission of the second conference participant terminal in the level data module to the conference module;

[0106] The conference module is connected to the feedback module, and is used to determine whether the playback permission of the second participant terminal includes the audio and video stream of the first participant terminal for the playback signaling, and send the judgment result to the feedback module;

[0107] The feedback module is connected to the level data module and the conference scheduling protocol proxy module, and is used to send the judgment result to the conference scheduling protocol proxy module when the judgment result is that the playback permission of the second conference participant terminal includes the audio and video stream of the first conference participant terminal;

[0108] The conference scheduling protocol proxy module is connected to the conference scheduling server and the level data module, and is used to generate a flow adjustment request for scheduling the audio and video stream of the first participant terminal to the second participant terminal according to the judgment result sent by the feedback module, and send the flow adjustment request to the conference scheduling server for flow adjustment.

[0109] In the embodiment of the present invention, a connection relationship between each module in the command and dispatch terminal is provided. Through the interaction between each module, the steps provided in the first and second embodiments of the present invention can be completed. Some of the steps mentioned in the first and second embodiments can be executed with reference to the corresponding modules. For example, if the command and dispatch terminal needs to send a message to the Web background service in the Web server, it can send it to the Web background service through the feedback module; if the protocol conversion and data forwarding with the CMS conference management service in the conference scheduling server are performed, it can be achieved through the conference scheduling protocol proxy module. The specific implementation principle and process of the embodiment of the present invention refer to the explanation of the first and second embodiments, and no further limitation is made here.

[0110] In an embodiment of the present invention, when the command and dispatch end receives a playback signal sent by the Web background service for playing the audio and video stream of the first participant terminal, not all second participant terminals have the authority to play the audio and video stream of the first participant terminal. The command and dispatch end will perform a screening based on the playback authority of each second participant terminal, and then dispatch the audio and video stream of the first participant terminal to the second participant terminal that has the playback authority of the audio and video stream of the first participant terminal. This can solve the problem of transmitting audio and video streams in the network under a hierarchical structure and improve the privacy and security of the video conferencing scenario.

[0111] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0112] Reference Figure 6 , corresponding to the method of embodiment 1, a schematic diagram of the structure of embodiment 3 of an audio and video stream scheduling device of the present invention is shown, which can be applied to a command and dispatch end. The device may specifically include the following modules:

[0113] The play signaling receiving module 601 is used to receive a play signaling sent by the Web server for playing the audio and video stream of the first conference participant terminal;

[0114] The first playback permission judgment module 602 is used to judge whether the playback permission of the second conference participant terminal stored in advance includes the audio and video stream of the first conference participant terminal according to the playback signaling;

[0115] The stream adjustment request generating module 603 is used to generate a stream adjustment request for scheduling the audio and video stream of the first participating terminal to the second participating terminal when the playback permission of the second participating terminal includes the audio and video stream of the first participating terminal;

[0116] The flow adjustment request sending module 604 is used to send the flow adjustment request to the conference scheduling server for flow adjustment.

[0117] In an optional embodiment of the present invention, the first conference participant terminal and the second conference participant terminal both have a conference participant level, and the conference participant levels of the plurality of second conference participant terminals and the first conference participant terminal are in a tree structure;

[0118] The playback authority of the second participating terminal is determined according to the participating level of the second participating terminal in the tree structure.

[0119] In an optional embodiment of the present invention, the first playback authority determination module 602 may include the following submodules:

[0120] a level relationship determination submodule, configured to determine, with respect to the play signaling, a relationship between the participation level of the second participating terminal and the participation level of the first participating terminal in the tree structure;

[0121] The audio and video stream judgment submodule is used to judge whether the pre-stored playback permission of the second participating terminal includes the audio and video stream of the first participating terminal according to the relationship between the participating level of the second participating terminal and the participating level of the first participating terminal in the tree structure.

[0122] In an optional embodiment of the present invention, the first conference participant terminal has a first terminal number, and the second conference participant terminal has a second terminal number; the flow adjustment request generation module 603 may include:

[0123] A terminal number acquisition submodule, used to acquire a first terminal number of the first participating terminal and a second terminal number of the second participating terminal when the playback permission of the second participating terminal includes the audio and video stream of the first participating terminal;

[0124] The stream adjustment request encapsulation submodule is used to encapsulate the first terminal number and the second terminal number into the stream adjustment request to schedule the audio and video stream of the first participating terminal to the second participating terminal.

[0125] In an optional embodiment of the present invention, the play signaling further includes a third terminal number of a third participant terminal for specifying the play of the audio and video stream of the first participant terminal; the device further includes:

[0126] A storage data judgment module, used for judging whether the playback permission of the third participating terminal is pre-stored for the third terminal number;

[0127] The second playback permission judgment module is used to judge whether the playback permission of the third participant terminal includes the audio and video stream of the first participant terminal when the playback permission of the third participant terminal is pre-stored.

[0128] In an optional embodiment of the present invention, the device may further include:

[0129] An error message generating module, used for generating an error message when the playback permission of the third conference participant terminal is not pre-stored;

[0130] The error information sending module is used to send the error information to the Web server.

[0131] According to another aspect of an embodiment of the present invention, there is also provided an audio and video stream scheduling device, including:

[0132] one or more processors; and

[0133] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the device to perform the above-mentioned audio and video stream scheduling method.

[0134] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein a computer program stored in the medium enables a processor to execute the above-mentioned audio and video stream scheduling method.

[0135] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0136] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0137] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0138] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0141] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0142] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0143] The above is a detailed introduction to an audio and video stream scheduling method, an audio and video stream scheduling device, a command scheduling terminal and a storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for scheduling audio and video streams, characterized in that: Applied to the command and dispatch end, the method includes: Receiving a play signaling sent by the Web server for playing the audio and video stream of the first conference participant terminal; For the play signaling, determine whether the pre-stored play permission of the second participating terminal includes the audio and video stream of the first participating terminal; the first participating terminal and the second participating terminal both have a participation level, and the participation levels of the plurality of second participating terminals and the first participating terminal are in a tree structure; wherein the play permission of the second participating terminal is determined according to the participation level of the second participating terminal in the tree structure; When the playback permission of the second participant terminal includes the audio and video stream of the first participant terminal, generating a stream modulation request for scheduling the audio and video stream of the first participant terminal to the second participant terminal; The flow adjustment request is sent to the conference scheduling server for flow adjustment.

2. The method according to claim 1, characterized in that For the play signaling, determining whether the pre-stored play permission of the second participant terminal includes the audio and video stream of the first participant terminal includes: For the play signaling, determining a relationship between a participation level of the second participating terminal and a participation level of the first participating terminal in the tree structure; According to the relationship between the participation level of the second participating terminal and the participation level of the first participating terminal in the tree structure, it is determined whether the pre-stored playback permission of the second participating terminal includes the audio and video stream of the first participating terminal.

3. The method according to any one of claims 1 to 2, characterized in that: The first conference participant terminal has a first terminal number, and the second conference participant terminal has a second terminal number; When the playback permission of the second conference participant terminal includes the audio and video stream of the first conference participant terminal, generating a stream modulation request for scheduling the audio and video stream of the first conference participant terminal to the second conference participant terminal includes: When the playback permission of the second participating terminal includes the audio and video stream of the first participating terminal, obtaining a first terminal number of the first participating terminal and a second terminal number of the second participating terminal; The first terminal number and the second terminal number are encapsulated as the stream modulation request to dispatch the audio and video stream of the first participant terminal to the second participant terminal.

4. The method according to claim 1, characterized in that The play signaling also includes a third terminal number of a third participant terminal for specifying the play of the audio and video stream of the first participant terminal; the method also includes: For the third terminal number, determining whether the playback permission of the third participating terminal is pre-stored; When the playback permission of the third participant terminal is pre-stored, it is determined whether the playback permission of the third participant terminal includes the audio and video stream of the first participant terminal.

5. The method according to claim 4, characterized in that The method further comprises: When the playback permission of the third conference participant terminal is not pre-stored, generating an error message; The error message is sent to the Web server.

6. A command and dispatch terminal, characterized in that: It includes a scheduling module, a conference module, a feedback module, a level data module and a conference scheduling protocol agent module, among which: The level data module is connected to the scheduling module and is used to store the playback authority of the second participant terminal; The scheduling module is connected to the Web server, the conference module and the conference scheduling protocol proxy module, and is used to send the playback signaling sent by the Web server for playing the audio and video stream of the first participant terminal to the conference module and the conference scheduling protocol proxy module, and is used to send the playback permission of the second participant terminal in the level data module to the conference module; the first participant terminal and the second participant terminal both have a participant level, and the participant levels of the plurality of second participant terminals and the first participant terminal are in a tree structure; wherein the playback permission of the second participant terminal is determined according to the participant level of the second participant terminal in the tree structure; The conference module is connected to the feedback module, and is used to determine whether the playback permission of the second participant terminal includes the audio and video stream of the first participant terminal for the playback signaling, and send the judgment result to the feedback module; The feedback module is connected to the level data module and the conference scheduling protocol proxy module, and is used to send the judgment result to the conference scheduling protocol proxy module when the judgment result is that the playback permission of the second conference participant terminal includes the audio and video stream of the first conference participant terminal; The conference scheduling protocol proxy module is connected to the conference scheduling server and the level data module, and is used to generate a flow adjustment request for scheduling the audio and video stream of the first participant terminal to the second participant terminal according to the judgment result sent by the feedback module, and send the flow adjustment request to the conference scheduling server for flow adjustment.

7. An audio and video stream scheduling device, characterized in that: Applied to the command and dispatch end, the device comprises: A play signaling receiving module, used to receive a play signaling sent by a Web server for playing the audio and video stream of the first conference participant terminal; a first playback permission judgment module, used to judge, for the playback signaling, whether the playback permission of the pre-stored second conference participant terminal includes the audio and video stream of the first conference participant terminal; the first conference participant terminal and the second conference participant terminal both have a conference participation level, and the conference participation levels of the plurality of second conference participants and the first conference participant terminal are in a tree structure; wherein the playback permission of the second conference participant terminal is determined according to the conference participation level of the second conference participant terminal in the tree structure; a stream adjustment request generating module, configured to generate a stream adjustment request for dispatching the audio and video stream of the first participating terminal to the second participating terminal when the playback permission of the second participating terminal includes the audio and video stream of the first participating terminal; The flow adjustment request sending module is used to send the flow adjustment request to the conference scheduling server for flow adjustment.

8. An audio and video stream scheduling device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the device to perform the audio and video stream scheduling method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The computer program stored therein enables the processor to execute the audio and video stream scheduling method as described in any one of claims 1 to 5.

Images