AV Conference Realization Method, AV Conference System, and Computer Program

Abstract

This application discloses an AV conference implementation method, an AV conference system and related devices, and this application relates to a plurality of device groups, each of which includes a plurality of interface devices, and each interface device connects a conference access terminal so that the conference access terminal accesses a conference room through the interface device. The conference access terminals are distributed to different interface devices, and the load is shared by the plurality of interface devices to realize the support of large-scale conferences. Each device group has a back-source device, which accesses the conference room through the second interface device to generate a media data stream, and the back-source device in each device group, for example, the first device group, transfers the media data stream corresponding to the first identification information to the first interface device after receiving the pull stream request of the first interface device in the first device group, and the first interface device transfers the media data stream to the corresponding conference access terminal. The back-source device shares the data distribution pressure, which facilitates the support of super-large-scale conferences, and can open microphones and videos at the same time, without the need for redirection, and the experience is smooth.

Description

Technical Field

[0001] This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on July 30, 2021, with the application number 202110875339.6 and the invention title "AV Conference Realization Method, AV Conference System and Related Devices", and all its contents are incorporated into this application by reference.

[0002] This application relates to the field of computers, and particularly to an AV conference realization method, an AV conference system and related devices.

Background Art

[0003] With the rapid development of network technology, communication technology and streaming media technology, and the increasing mobility of people's work and learning, the needs of enterprises and individuals for video communication are also increasing, and AV conference systems have emerged.

[0004] Conventional AV conferences (also called voice and video conferences) often adopt a conference architecture based on a Selective Forwarding Unit (SFU), which is composed of one server and multiple terminals. After the server receives the AV stream (i.e., media data stream) shared by a certain terminal in the conference room, it directly forwards the AV stream to other terminals in the conference room.

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in this conference architecture, if the number of people entering a conference room is too large, it will greatly increase the data distribution pressure on the server. Therefore, in this conference architecture, the number of participants in a conference is limited, and at the same time, the number of people who can turn on the microphone and video is limited. If a general viewer wants to raise a hand to speak, they need to be redirected, and the experience is not very smooth.

Means for Solving the Problems

[0006] To solve the above technical problems, the present application provides an AV conference implementation method, an AV conference system, and related devices that can significantly reduce the data distribution pressure of media source devices such as a second interface machine. Therefore, it can support a larger number of participants, and can even support the participation of millions of people. At the same time, since the data distribution pressure during the transmission of the media data stream is significantly reduced, all participants can open their microphones and videos simultaneously without redirection, and the experience is smooth.

[0007] Embodiments of the present application disclose the following technical solutions.

[0008] In a first aspect, embodiments of the present application provide an AV conference implementation method, and the method includes: A back-to-source device in a first device group receives a pull stream request of a first interface machine in the first device group, where the pull stream request includes first identification information of a media data stream. The first device group is any one of a plurality of device groups, and there is a back-to-source device in each device group of the plurality of device groups. Each device group includes a plurality of interface machines. Each interface machine of the plurality of interface machines is used to connect a conference access terminal so that the conference access terminal can access the conference room through the interface machine. The first interface machine is any one in the first device group. The back source device in the first device group transfers the media data stream corresponding to the first identification information to the first interface device so that the first interface device transfers the media data stream to the corresponding conference access terminal. The media data stream is obtained by the back source device in the first device group from the second interface device in the second device group, and the conference access terminal that generates the media data stream accesses the conference room through the second interface device. The second device group is any one of the plurality of device groups. This step includes:

[0009] In a second aspect, the embodiments of the present application provide an AV conference implementation device, and the device includes: A receiving unit used to receive a pull stream request of a first interface device in a first device group. The pull stream request includes first identification information of a media data stream. The first device group is any one of a plurality of device groups. Each device group of the plurality of device groups has a back source device, and each device group includes a plurality of interface devices. Each of the plurality of interface devices is used to connect a conference access terminal so that the conference access terminal accesses the conference room through the interface device. The first interface device is any one of the first device group. The receiving unit; A transfer unit used to transfer the media data stream corresponding to the first identification information to the first interface device so that the first interface device transfers the media data stream to a conference access terminal corresponding to the media data stream. The media data stream is obtained from a second interface device in a second device group by a back-source device in the first device group. The conference access terminal that generates the media data stream accesses the conference room through the second interface device, and the second device group is any one of the plurality of device groups. The transfer unit includes:

[0010] In a third aspect, an embodiment of the present application provides an AV conference system, and the system includes a data transmission network and a room management subsystem. The data transmission network is used to transmit a media data stream generated in a conference room. The data transmission network includes a plurality of device groups, and each device group includes a plurality of interface devices. Each of the plurality of interface devices is used to connect a conference access terminal so that the conference access terminal accesses the conference room through the interface device. Among the plurality of device groups, there is a back source device in each device group. The back source device in the first device group receives a pulse stream request from the first interface machine in the first device group. The pulse stream request includes the first identification information of the media data stream. It is used to transfer the media data stream corresponding to the first identification information to the first interface machine. The media data stream is obtained by the back source device in the first device group from the second interface machine in the second device group. The conference access terminal that generates the media data stream accesses the conference room through the second interface machine. The first device group and the second device group are each one of the plurality of device groups. The first interface machine is any one of the interface machines in the first device group. The first interface machine is used to transfer the media data stream to the corresponding conference access terminal. The room management subsystem includes the plurality of device groups and a center server. Each device group includes an in-group server. The in-group server is used to assist the center server in managing the conference room.

[0011] In a fourth aspect, an embodiment of the present application provides an electronic device for realizing an AV conference. The electronic device includes a processor and a memory. The memory stores program code and is used to transmit the program code to the processor. The processor is used to execute the method described in the first aspect according to the commands in the program code.

[0012] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store program code, and when the program code is executed on an electronic device, the electronic device is caused to execute the method described in the first aspect.

[0013] In a sixth aspect, an embodiment of the present application provides a computer program, and when the computer program is executed, the electronic device is caused to execute the method described in the first aspect.

[0014] According to the above technical solution, the present application relates to a plurality of device groups, each device group includes a plurality of interface devices, and each interface device of the plurality of interface devices is used to connect a conference access terminal so that the conference access terminal can access the conference room through the interface device. Compared with the related art in which a plurality of conference access terminals access a conference room through one server, the plurality of interface devices play a role similar to that of the server in the related art, distribute the conference access terminals to different interface devices, and share the load by the plurality of interface devices, so that it is easier to access more conference access terminals and support large-scale conferences. There is a back-source device in each device group among the plurality of device groups, which accesses the conference room through the second interface device to generate (produce) a media data stream. The back-source device in each device group, for example, the first device group, after receiving the pull stream request of the first interface device in the first device group, since the pull stream request includes the first identification information of the media data stream, can transfer the media data stream corresponding to the first identification information to the first interface device, and the first interface device is used to transfer the media data stream to the corresponding conference access terminal. In this way, when each interface device needs to pull the media data stream of the second interface device, the second interface device only needs to interact with the back-source device in each individual device group. Furthermore, the back-source device shares the data distribution pressure of the media source device, for example, the second interface device, and does not need to interact with other interface devices. The second interface device does not need to interact with each interface device, greatly reducing the data distribution pressure of the device, for example, the second interface device.Therefore, the method for realizing the AV conference can support a larger number of participants, and can even support the participation of millions of people. At the same time, since the data distribution pressure during the transmission of the media data stream is greatly reduced, all participants (i.e., users) can turn on the microphone and video simultaneously without redirection, and the experience is smooth.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4a

Figure 4b

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0016] To more clearly explain the technical solutions in the embodiments of this application or in the prior art, the drawings necessary for the description of the embodiments or the prior art are briefly introduced below. Obviously, the drawings in the following description are only a part of the embodiments of this application. For those skilled in the art, if no creative labor is required for progressiveness, other drawings can also be obtained according to these drawings. The embodiments of this application will be described below in conjunction with the drawings.

[0017] Traditional AV conferences often adopt a conference architecture based on SFU. Referring to Figure 1, in the drawing, the black nodes are servers, the gray nodes are terminals, multiple terminals are connected to the server, and users A, B, and C access the conference room through their corresponding terminals and participate in the conference.

[0018] However, in this conference architecture, if the number of people entering a single conference room is too large, it will greatly increase the data distribution pressure on the server. Therefore, in this conference architecture, the number of participants in a single conference is limited, and at the same time, the number of people who can turn on the microphone and video simultaneously is limited. When a general viewer tries to raise their hand to speak, they need to be redirected, and the experience is not very smooth.

[0019] Also, in this method, the participants cannot turn on the video and microphone simultaneously. Therefore, based on the SFU architecture shown in Figure 1, diffusion agents (shown as the nodes included in the dashed block in Figure 2) are increased, and the users who do not speak are placed as diffusion agents. When raising their hands to speak, they are redirected to the call nodes (shown as the nodes included in the solid block in Figure 2), and the distribution advantage of the diffusion agents is utilized to expand the number of conference participants. However, this method requires redirection, and the experience is not very smooth.

[0020] To solve the above technical problem, an embodiment of the present application provides an AV conference implementation method. According to this method, when each interface device needs to pull the media data stream of the second interface device, the second interface device only needs to interact with the back source devices in the individual device group. Furthermore, the back source devices share the data distribution pressure of the second interface device, interact with other interface devices, and the second interface device does not need to interact with each interface device, greatly reducing the data distribution pressure of the media source device, such as the second interface device. Therefore, the AV conference implementation method can support a larger number of participants, and can even support the participation of millions of people. At the same time, since the data distribution pressure during the transmission of the media data stream is greatly reduced, all participants do not need to be redirected and can open the microphone and video simultaneously, providing a smooth experience.

[0021] It should be noted that the AV conference implementation method provided by the embodiments of the present application can be applied to various AV conference scenarios, especially for use in ultra-large-scale AV conferences with millions of participants in a single conference.

[0022] Next, the AV conference system architecture provided by the embodiments of the present application will be introduced. When participants enter the conference room to participate in the conference based on the AV conference system, the participants mainly include guests who need to turn on the AV and general viewers. When the guest turns on the AV, a media data stream is generated, and the media data stream is transmitted to the conference access terminals corresponding to the general viewers and other guests. Here, the conference access terminal is the terminal used when accessing the conference room, and the conference access terminal may be a user terminal. Therefore, in the AV conference, mainly when the participants turn on the AV, such as a video or a microphone, there is the transmission of the media data stream. If the transmission of the media data stream is not limited by the number of participants in the conference room, the AV conference system can support large-scale conferences. Therefore, the focus of the embodiments of the present application lies in the improvement of the data transmission network architecture.

[0023] Also, each individual conference corresponds to one conference room, saves the room information and the participation list, and manages the user state (turn on or off the AV, that is, turn on or off the microphone, turn on or off the video, shoot, enter or exit the room, etc.). In the case of an ultra-large-scale conference, all participants constantly turn on or off the video, or enter or exit the room, generating a large number of notification messages representing state changes, and the pull and synchronization of the participation list of the large-scale AV conference also impose great pressure on the room management subsystem. Therefore, the improvement to the room management subsystem in the embodiments of the present application also enables the AV conference system to support ultra-large-scale AV conferences.

[0024] Therefore, in this embodiment, as shown in FIG. 3, the AV conference system architecture mainly includes a data transmission network 301 and a room management subsystem 302. The data transmission network 301 is used to transmit the media data stream generated in the conference room, and the room management subsystem 302 is used to manage the conference room.

[0025] First, from the perspective of the data transmission network 301 transmitting a media data stream, embodiments of the present application introduce an AV conference implementation method. Referring to FIG. 4a, the method includes the following content.

[0026] In S401, the back-source device in the first device group receives a pulse stream request from the first interface machine in the first device group.

[0027] As shown in FIG. 4b, the data transmission network 301 includes a plurality of device groups 401, for example, each shown by a dashed block in FIG. 4b. Each device group 401 includes a plurality of interface machines, and each interface machine is, for example, the node shown by a circle in the device group 401 in FIG. 4b. Each of the plurality of interface machines is used to connect a conference access terminal, so that the conference access terminal can access the conference room through the interface machine, and the participating personnel corresponding to the conference access terminal can enter the conference room to participate in the conference. Here, the device group may be indicated by SET, that is, the plurality of interface machines are partitioned according to SET, and the interface machine may be a server. Each of the plurality of device groups 401 has a back-source device.

[0028] Here, the first device group may be any one of the plurality of device groups, and the first interface machine may be any one of the interface machines in the first device group.

[0029] When a certain interface device (e.g., the first interface device) in a certain device group (e.g., the first device group) needs to pull a media data stream, the first interface device may send a pull stream request to a back source device in the first device group. The pull stream request may include first identification information of the media data stream, and the first identification information identifies the media data stream of the subscribed media data, so that the back source device can know which media data stream needs to be pulled. Here, the first interface device may be an interface device connected to a conference access terminal corresponding to a user who has subscribed to media data. That is, when a user in the first interface device subscribes to the media data of a certain interface device, it triggers the first interface device to send a pull stream request to the back source device in the first device group.

[0030] In S402, the back source device in the first device group transfers the media data stream corresponding to the first identification information to the first interface device, so that the first interface device transfers the media data stream to the corresponding conference access terminal.

[0031] After the back source device in the first device group receives the pull stream request, it transfers the media data stream corresponding to the first identification information to the first interface device, and the first interface device transfers the media data stream to the corresponding conference access terminal. Here, the conference access terminal that generates the media data stream accesses the conference room through the second interface device, and the second device group is any one of the plurality of device groups.

[0032] The second interface device may be an interface device connected to a conference access terminal corresponding to a user who has turned on AV. For example, when a certain user, for example, the second user, turns on the video on the conference access terminal, the conference access terminal corresponding to the second user accesses the conference via the second interface device, and the second interface device is in the second device group. When any one of the users who want to participate in the conference, for example, the first user, wants to view the video of the second user (for example, subscribe to the video of the second user), the first interface device connected to the conference access terminal corresponding to the first user may find the back-source device in the device group to which it belongs, for example, the first device group, and send a pull stream request to the back-source device in the first device group. As a result, the back-source device transfers the media data stream obtained from the second interface device in response to the pull stream request to the first interface device, and the first interface device transfers the media data stream to the corresponding conference access terminal. Here, the user related to the embodiment of the present application may be a participant.

[0033] In addition, the media data stream in the back-source device may be pulled from the second interface device, and the pulled media data stream may be stored locally in the back-source device. That is, generally, when the back-source device receives a pull stream request to pull the media data stream for the first time, since the media data stream is not stored locally, it is necessary to pull it from the second interface device. If another interface device has requested it before, the media data stream is stored locally and there is no need to pull it from the second interface device.

[0034] Therefore, in this embodiment, different media data stream pulling methods may be adopted according to whether the back-source device in the first device group searches and finds the media data stream locally.

[0035] In one possible embodiment, a specific way for the back-source device in the first device group to transfer the media data stream corresponding to the first identification information to the first interface device is that the back-source device in the first device group receives the pull stream request sent from the first interface device, searches for the media data stream of the corresponding second interface device according to the first identification information, and if the media data stream of the second interface device cannot be found after searching, transfers the pull stream request to the second interface device, receives the media data stream transferred according to the first identification information in the pull stream request by the second interface device, and transfers the media data stream to the first interface device. This may also be the case.

[0036] As can be understood, when a pull stream request is sent when the first user subscribes to the video of the second user, after receiving the pull stream request, the second interface device may record that the back-source device has subscribed to the second interface device locally at the second interface device, and then the second interface device sends the media data stream to the back-source device. The back-source device stores the received media data stream locally, so that when other subsequent interface devices request the media data stream, the back-source device does not need to interact with the second interface device and may directly return the media data stream to the first interface device.

[0037] For example, as shown in FIG. 4b, here, node A may be the second interface device, and the second interface device is the interface device where the guest is located. That is, the user corresponding to the conference access terminal in the second interface device may turn on the video, upload the video data stream, and at this time, the video data stream is used as the media data stream. Node B may be the first interface device, and node D may be the back source device of the device group where it is located, for example, the first device group. The user at node B subscribes to the upstream video data stream of node A. Node B finds node D within the first device group and issues a pull stream request to node D. After node D receives the pull stream request from node B, it searches whether there is a media data stream of node A locally. If it cannot find the media data stream of node A locally, it issues a pull stream request to node A. After node A receives the pull stream request from node D, it records locally that node D has subscribed to node A, transfers the video data stream to node D, and after node D receives the video data stream, it transfers it to node B. Node B finally sends the video data stream to the corresponding conference access terminal.

[0038] When the back source device in the first device group searches for the media data stream of the second interface device according to the first identification information, if the media data stream of the second interface device is found through the search, the directly obtained media data stream is transferred to the first interface device. By this method, the pull efficiency of the media data stream can be improved, and furthermore, the data distribution pressure of the media source device, for example, the second interface device, can be reduced.

[0039] For example, as shown in FIG. 4b, here, node A may be the second interface device, and the second interface device is the interface device where the guest is located. That is, the user corresponding to the conference access terminal in the second interface device may turn on the video, upload the video data stream, and at this time, the video data stream becomes the media data stream. Node C may be the first interface device, and node D may be the back-source device of the device group to which it belongs, for example, the first device group. The user at node C subscribes to the upstream video data stream of node A. Node C finds node D within the first device group and issues a pull stream request to node D. Nodes B and C in the same device group both find the same node D by means of consistent hashing or another method and issue a pull stream request to node D. Node D searches locally and finds that there is this video data stream (requested by node B before). Node D directly transfers the video data stream obtained by the search to node C. Node C receives the video data stream transferred from node D and finally transfers the video data stream to the corresponding conference access terminal.

[0040] As can be understood, the back-source device in the first device group is obtained by being specified according to a preset rule by an interface device that needs to request a media data stream, for example, the first interface device. The preset rule is such that the same media data stream is routed to the interface device in the device group to which it belongs via the same back-source device. For example, nodes B and C in the same device group both find the same node D by means of consistent hashing or another method. Thereby, it becomes easy to perform centralized distribution management for the same media data stream in the same device group.

[0041] Note that the first device group and the second device group may be located in the same local area network (i.e., the internal network), or may be located in different local area networks (i.e., the external network, including the external networks of domestic networks and foreign networks). When the first device group and the second device group are located in the same local area network, the back source device in the first device group directly transfers the pull stream request to the second interface machine, and the second interface machine directly sends the media data stream to the back source device. When the first device group and the second device group are located in different local area networks, the data transmission network 301 further includes a relay agent device 402, for example, the device shown as 402 in FIG. 4b. The back source device in the first device group needs to transfer the pull stream request to the second interface machine via the relay agent device 402. Correspondingly, the second interface machine transfers the media data stream to the back source device in the first device group via the relay agent device 402.

[0042] For example, as shown in FIG. 4b, here, node A may be the second interface device, and the second interface device is the interface device where the guest is located. That is, the user corresponding to the conference access terminal in the second interface device may turn on the video, upload the video data stream, and at this time, the video data stream is used as the media data stream. Node G may be the first interface device, and node F may be the back-source device of the device group it belongs to, for example, the first device group. At this time, the first device group where node G is located and the second device group where node A is located are not located in the same local area network, that is, node G is the edge node. The user at node G subscribes to the upstream video data stream of node A (the video data stream is used as the media data stream) and obtains the video data stream. The path is relatively long. Node G finds node F in the first device group and issues a pull stream request to node F. After node F receives the pull stream request from node G, it searches locally whether there is a video data stream of node A. If it cannot be found after searching, it issues a pull stream request to node A through the relay agent device 402 (for example, node E in FIG. 4b). After node A receives the pull stream request from node F, it records locally that node F has subscribed to node A, and then transfers the video data stream to node F through node E. Node F transfers the video data stream to node G, and node G finally transfers the video data stream to the corresponding conference access terminal.

[0043] The process in which node H in FIG. 4b pulls the video data stream of node A is similar to that of node C, except that it also passes through node E on the way, and the repeated description is omitted here. In this process, the search for the relay agent device 402, for example, node E, may be preset or dynamically allocated, and is not restricted here. Structurally, the embodiments of the present application partition the interface machine for accessing the AV conference into a device group (SET). When the interface machine in a certain device group needs to pull the media data stream, the back-source device is identified in the device group where it is located, so that a pull stream request may be sent to the interface machine (i.e., the media source device) that generates the media data stream by the back-source device. This design greatly reduces the data distribution pressure of the media source device and greatly improves the number of participants in the conference room. For example, if each SET has 100 interface machines, the entire AV conference system has 100 SETs, and it is assumed that each 8-core 16G interface machine can support the transfer of 100 paths of media data streams. If a conference has only one guest and the others are viewers, the maximum scale of this conference is 100 (units) * 100 (SETs) * 100 (deliveries) = 1 million people. If there is more than one guest, based on the principle of nearest access, different interface machines in different SETs are allocated, and the uplink transfer load is also dispersed, making it easy to support a conference of one million people.

[0044] According to the above technical solution, the present application relates to a plurality of device groups, each device group includes a plurality of interface devices, and each interface device among the plurality of interface devices is used to connect a conference access terminal so that the conference access terminal can access the conference room through the interface device. Compared with the related technology in which a plurality of conference access terminals access a conference room through one server, the plurality of interface devices play a role similar to that of the server in the related technology, distribute the conference access terminals to different interface devices, and share the load by the plurality of interface devices, so that it is easier to access more conference access terminals and support large-scale conferences. There is a back-source device for each device group among the plurality of device groups. The back-source device accesses the conference room through the second interface device to generate a media data stream. The back-source device in each device group, for example, the first device group, after receiving the pull stream request of the first interface device in the first device group, since the pull stream request includes the first identification information of the media data stream, it can transfer the media data stream corresponding to the first identification information to the first interface device, and the first interface device is used to transfer the media data stream to the corresponding conference access terminal. In this way, when each interface device needs to pull the media data stream of the second interface device, the second interface device only needs to interact with the back-source device in each individual device group. Furthermore, the back-source device shares the data distribution pressure of the media source device, for example, the second interface device, and does not need to interact with other interface devices. The second interface device does not need to interact with each interface device, greatly reducing the data distribution pressure of the device, for example, the second interface device.Therefore, the method for realizing the AV conference can support a larger number of participants, and can even support the participation of millions of people. At the same time, since the data distribution pressure during the transmission of the media data stream is greatly reduced, all participants (i.e., users) can turn on the microphone and video simultaneously without redirection, and the experience is smooth.

[0045] Compared with the conference architecture based on a Multipoint Control Unit (MCU), the operation and maintenance are simpler, not restricted by performance and hardware price, greatly improving the actual bearing number, and it is easy to realize an ultra-large-scale conference.

[0046] For the interface diagram of conducting a video conference based on the method and system for realizing an AV conference provided by the embodiments of the present application, reference can be made to FIG. 5. In this interface, images of participants with the video turned on, a participation list, and icons of related function keys (such as a microphone icon, a camera icon, a management member icon, an icon for exiting the conference, etc.) can be displayed. For participants who turn on the microphone, the microphone icon can be displayed, and for participants who do not turn on the microphone, a slash can be drawn on the microphone icon. In the AV conference system provided by the embodiments of the present application, an ultra-large-scale conference can be realized. For example, the upper limit of the number of people entering the conference may be 50,000, that is, 50,000 management members.

[0047] Next, the embodiments of the present application introduce the method for realizing an AV conference from the perspective of the room management subsystem 302 managing the conference room.

[0048] As shown in FIG. 6, the room management subsystem 302 includes the above-mentioned plurality of device groups 601 and the center server (RoomSvc) 602. Here, the device group 601 also includes the above-mentioned plurality of interface machines, which is similar to the device group shown in FIG. 4b. At the same time, each device group 601 further includes an in-group server (RoomSvcInSet). The in-group server is used to assist the center server 602 in managing the conference room. Here, the in-group servers are, for example, node B, node C, node D, node E, node F, and node G in each device group 601 in FIG. 6, and the center server 602 is, for example, node A in FIG. 6.

[0049] It should be noted that in this embodiment, the in-group server and the back-source device in the same device group may be the same device or different devices, and this embodiment does not limit this.

[0050] Managing the conference room mainly includes managing the room information of the conference room, the notification message of the status change, the pull of the participation list, etc. Here, the room information includes, for example, the name of the conference room, the creation time, the participants, the meeting time, etc.

[0051] In the embodiment of the present application, the room management system is divided into a center server and in-group servers. There is only one center server, and the in-group servers are arranged according to the SET in the same way as the interface machines. Thereby, both the synchronization of the notification message and the pull of the participation list may be performed by the in-group servers to relieve the management pressure of the center server, and it is easy to realize an ultra-large-scale conference.

[0052] Inside the device group, for the same conference room, all interface devices register the same RoomSvcInSet according to a certain algorithm, maintain one long connection through heartbeats, and then pull the media data stream in the conference room. When there is a notification message that requires synchronization in the conference room (such as entering the conference room to participate in the meeting, leaving the conference room, turning on or off the microphone, turning on or off the video, etc., user status changes), or when wanting to pull the participation list, this is completed by this RoomSvcInSet.

[0053] First, the registration process for entering the conference room and participating in the meeting will be introduced. When any one interface device in any one device group, for example, the third interface device in the third device group, requests to enter the conference room, the in-group server in the third device group receives the registration request sent from the third interface device. The registration request may include a room identifier used to indicate which conference room needs to be entered. If the in-group server in the third device group fails to locally search for the information of the conference room and cannot find it, it sends the registration request to the center server. The center server records the second identification information of the in-group server in the third device group in the first meeting registration list and returns the registration success information to the in-group server in the third device group. The in-group server in the third device group records the third identification information of the third interface device in the second meeting registration list and returns the registration success information to the third interface device, thereby completing the registration of the third interface device. Here, the second identification information and the third identification information are of the same type of identification information and are the unique identifiers of the corresponding devices. The identification information may be, for example, an address. That is, the second identification information may be the address of the in-group server, and the third identification information may be the address of the third interface device. Here, the third device group is any one of the plurality of device groups, the third interface device is any one of the interface devices in the third device group, the third device group may be the first device group or the second device group in the foregoing embodiment, and the third interface device may be the first interface device or the second interface device in the foregoing embodiment.

[0054] For example, as shown in FIG. 6, here, node a may be a third interface device, and the third interface device is the interface device that wants to register a meeting. That is, the user corresponding to the meeting access terminal in the third interface device wants to enter the meeting room and participate in the meeting. Node A may be a central server, and node B may be a server within a device group where the third interface device is located, for example, a server within the third device group. Here, node B may be specified by node a according to a certain rule (consistent hash or allocation schedule). Node a sends a registration request to node B. After node B receives the registration request from node a, it searches the Scheduler for the RoomSvc corresponding to the meeting and finds that it is node A. Node B sends a registration request to node A. After node A receives the registration request sent from node B, it records the address of node B in the registration list of the meeting (the first meeting registration list stored in node A) and returns registration success information to B. After node B receives the registration success information returned from node A, it records the address of node a in the registration list of the meeting (the second meeting registration list stored in node B) and returns registration success information to node a.

[0055] As can be understood, in this embodiment, after successful registration, the server within the group, the central server, the server within the group and the successfully registered interface device, for example, the third interface device, can maintain a long connection, which facilitates subsequent interaction between the two. For example, node B may periodically send a heartbeat to node A to maintain a long connection channel with node A. After node a receives the registration success information returned from node B, it periodically sends a heartbeat to maintain a long connection channel with node B.

[0056] When the server within the group in the third device group searches locally for meeting room information in response to a registration request and finds the meeting room information locally, it may record the third identification information of the third interface device in the second meeting registration list and return registration success information to the third interface device.

[0057] For example, as shown in FIG. 6, here, node b may be a third interface machine. The third interface machine is an interface machine that wants to register for a meeting. That is, the user corresponding to the meeting access terminal in the third interface machine wants to enter the meeting room and participate in the meeting. Node A may be a central server, and node B may be a server within a device group where the third interface machine is located, for example, a server within the third device group. Here, node B may be specified by node b according to certain rules (consistent hash or allocation schedule). Node b sends a registration request to node B. Node B receives the registration request of node b, discovers that the information of the meeting room exists locally, records the address of node b in the registration list of the meeting (the second meeting registration list stored in node B), and returns registration success information to node b. After node b receives the registration success information returned from node B, it periodically sends a heartbeat to maintain a long connection channel with node B.

[0058] Similarly, when a user in an edge node (that is, the device group and the central server are not in the same local area network, and the interface machine in the device group and the central server are not in the same local area network) registers for the same meeting room, for the registration process, it is only necessary to relay through a relay agent device, for example, nodes O, P, and Q in FIG. 6, and the other processes are similar.

[0059] When the state of a meeting access terminal accessing a meeting room is changed, for example, when a user turns AV on or off, it is necessary to notify other users in the meeting room of this state change of turning AV on or off. Taking the case where the meeting access terminal with the state change is in the third interface device in the third device group as an example, at this time, the in-group server in the third device group receives the notification message sent from the third interface device and sends the notification message to the center server. The center server sends the notification message to the in-group servers of each device group, and the in-group servers of each device group send the notification message to the interface devices that have been successfully registered.

[0060] Subsequently, taking FIG. 6 above as an example, here, node a may be the third interface device, node a is the interface device where the guest is located, node A may be the center server, and node B may be the in-group server of the device group where the third interface device is located, for example, the third device group. When the guest at node a turns on the video, that is, when the state is changed, node a sends a notification message (msg-a-openvideo) indicating the state change to node B, and node B sends the notification message to node A. Node A sends the notification message to all in-group servers across the local registration list (the first meeting registration list stored in node A), for example, nodes B, C, D, E, F, G in FIG. 6. Nodes B, C, D, E, F, G receive the notification message and send the notification message to all interface devices across the local registration list (as the second meeting registration list stored in each in-group server), for example, nodes a, b, c, d, e, f, g, o, p, q, r, s in FIG. 6.

[0061] In an AV meeting, a participation list is served. The participation list includes all the participants. In a large-scale AV meeting realized by the AV meeting system provided by the embodiments of the present application, since the number of participants is huge, the method of paging pull can be adopted to update the local participation list. When an interface device in a certain device group, for example, the third interface device in the third device group, requests a pull of the participation list, the in-group server in the third device group receives the list pull request of the third interface device. The in-group server in the third device group searches for the participation list locally. If the in-group server in the third device group fails to find the participation list locally, it sends the list pull request to the center server. The in-group server in the third device group receives the participation list sent from the center server and sends the participation list to the third interface device.

[0062] For example, as shown in FIG. 6, taking node a as the third interface device, a user at node a requests a participation list from node B. Node B searches in its local cache. If it fails to find the list, it issues a list pull request to node A. After receiving the list pull request, node A returns the corresponding participation list to node B. After node B caches the participation list locally, it returns the participation list to node a. Here, the participation list may be the entire participation list, or it may be the participation list of a certain page with paging pull, such as Page1.

[0063] When the in-group server in the third device group searches for the participation list locally, if the in-group server in the third device group finds the participation list locally and the expiration date of the participation list has not expired, it returns the participation list to the third interface device.

[0064] Subsequently, according to the above example, taking node b shown in FIG. 6 as the third interface device, even if there were other nodes before it at node b, the above participation list Page1 was requested. At this time, since the participation list already exists at node B, there is no need for node A to request the participation list. Therefore, node a sends a list pull request to node B. Node B searches in the local cache, retrieves Page1, and the expiration date has not expired. Then, node B directly returns Page1 to node b.

[0065] Next, the disaster recovery process will be described. Since all the information of RoomSvc is obtained from RoomSvcInSet and the interface device, it is easy to be automatically reconstructed by RoomSvcInSet after RoomSvc goes down. Similarly, since the data of RoomSvcInSet is obtained from the interface device, it can be automatically reconstructed by the interface device within this device group (SET) after RoomSvcInSet goes down.

[0066] In the AV conference realization method provided by the embodiments of this application, the management pressure on the center server is relieved by the in-group servers, making it easy to realize ultra-large-scale conferences.

[0067] In the AV conference based on the SFU provided by the related technology, when a media data stream, such as an audio data stream, is generated in the transmission conference room, it is necessary to perform audio routing based on the energy value. In a long-distance call, the delay generated by this method has a great impact on the experience. When the number of participants is relatively large, a large amount of bandwidth is wasted, increasing the network performance loss of the conference access terminal. In addition, in the case of playing back Interactive Voice Response (IVR) in the conference and in a scene where there are requirements for accurate recording, it is not easy to perform accurate control (such as start and end judgment, whether by signaling or by media data stream, etc.) with the SFU configuration. In the case of voice recording, some industries with strict requirements cannot accept the situation of recording more or less a few words during the recording process. Without a centralized audio processing service, it is easy to record content that should not be recorded or not record content that should be recorded. These situations are absolutely unacceptable to strict users.

[0068] As shown in FIG. 3, the AV conference system provided by the embodiment of the present application further includes a mixing engine 303. The mixing engine 303 is between the media transmission network 301 and the Public Switched Telephone Network (PSTN) 305. As a bridge for both connections, it greatly improves the opening experience in a very large room. Here, the mixing engine 303 is connected to the public switched telephone network 305 via a protocol, such as the Session Initiation Protocol (SIP) or the Real-time Transport Protocol (RTP).

[0069] Here, the mixing engine 303 includes a mixer and a selector, and each device group has a corresponding mixing engine. When the media data stream is an audio data stream, the selector in the mixing engine sends the acquired multi-audio data stream to the mixer in the same device group, and the multi-audio data stream is from the fourth interface machine in different device groups. The mixer selects a target audio data stream from the multi-audio data stream and sends a winning notice to the target interface machine connected to the conference access terminal that generates the target audio data stream. The target interface machine sends the target audio data stream to the mixer, and the mixer mixes the target audio data stream and transfers the mixed audio data stream to the selector corresponding to the fourth interface machine. The fourth interface machine includes the target interface machine. The selector corresponding to the fourth interface machine sends the mixed audio data stream to the fourth interface machine, and the fourth interface machine transfers the mixed audio data stream to other interface machines in the device group to which it belongs.

[0070] The Mixer and Selector are also arranged by the SET. To reduce the delay as much as possible, the Mixer and Selector of one SET are arranged in the same available area (the transmission delay inside the available area may be controlled to 2 ms). At the same time, the audio data streams of the same conference are concentrated and processed by one SET, and finally concentrated and mixed by the same Mixer.

[0071] Assume that Figure 7 represents an actual meeting. Here, three participants turn on their microphones. The interface devices to which these three participants belong are Node a, Node f, and Node g, and the other interface devices are viewers. To simplify the example diagram, assume that each meeting selects two - path audio data streams (usually, a real meeting selects 4 - 6 - path audio data streams). At this time, the processing flow of the audio data stream is as follows.

[0072] Node a, Node f, and Node g request a selector from the scheduling system in Figure 6 (to minimize the delay, the scheduling system assigns selectors in the same available area for the audio data streams of the same meeting room), and obtain the selectors shown by Node B, Node C, and Node D in Figure 7 respectively. Node a, Node f, and Node g transfer the audio data streams to the corresponding Node B, Node C, and Node D respectively. Node B, Node C, and Node D cache the upstream audio data streams and upload the energy values to the mixer. The mixer sorts the energy values, then selects the two - path audio data streams of Node f and Node g as the target audio data streams, and notifies Node C and Node D that Node C and Node D are the target interface devices. After Node C and Node D receive the winning notice, they transfer the corresponding target audio data streams to the mixer (the delay of these two steps is within 5 ms). The mixer mixes the received target audio data streams, and then transfers the mixed audio data streams to Node B, Node C, and Node D. Node B, Node C, and Node D send the mixed audio data streams to Node a, Node f, and Node g. Node a, Node f, and Node g perform internal transfers within the SET.

[0073] Note that here, the stream sent to the guest is different from the stream sent to the general audience. The mixed audio data stream sent to the guest excludes the self-talking path. Therefore, the selected interface device receives the audio data streams of two paths, one of which is transferred to the speaking guest, and the other is transferred to the audience within the SET.

[0074] The mixing engine 303 is a real-time media processing system, and the audio data stream processed by the mixing engine 303 needs to be transferred to all the participants in the conference room in real time. In many cases, an asynchronous stream processing system may be required, that is, this system does not need to return to the participants in the conference room after obtaining the audio data stream. For other purposes (such as recording, pornographic identification, live streaming, etc.), usually, less real-time is required. Based on this, the AV conference system provided by the embodiments of the present application further includes a bypass media processing system 304 as shown in FIG. 3. The bypass media processing system 304 pulls the media data stream generated in the conference room by the robot terminal and is used to process the media data stream. Here, the robot terminal includes a live stream robot, a record robot, and an MRA robot as shown in FIG. 3.

[0075] In this embodiment, the bypass media processing function is realized by adopting the method that the Robot terminal enters the conference room and pulls the stream. Taking recording as an example, the user business background starts a recording task in the way of Application Programming Interface (API). The recording task system instantiates a Record Robot to simulate participants and enter the conference. The Record Robot pulls the media data stream in the conference locally, and performs recording after mixing and transcoding. After the conference ends, the Record Robot uploads the recorded file to the specified storage (it is also possible to upload some fragments regularly).

[0076] Note that the identification of porn and the solution of live push streaming are similar to recording. The difference is only the business logic. When accessing a third-party conference device, in order to perform two-way communication, it is slightly different from the recording solution.

[0077] When accessing a third-party conference device, as shown in FIG. 8, the user business background starts an MRA (conference connector) Robot. The MRA Robot enters the conference (at this time, other participants can see this MRA Robot). At the same time, the MRA Robot is connected to the third-party conference device by a connection protocol (such as protocols like SIP or H323). The MRA Robot pulls all the media data streams in the conference locally, mixes them, and then transfers them to the third-party conference device by the connection protocol. The MRA Robot converts the media data stream sent from the third-party conference device into the private protocol of the conference and transfers it to other participants in the conference room through the media transmission network. Here, 801 in FIG. 8 is a mobile conference terminal, 802 is the intelligent conference room of the conference room, and it is used to access the conference.

[0078] In addition, when the AV conferencing system does not introduce a mixing engine, the connection between the data transmission network 301 and the public switched telephone network 305 may be realized using the architecture shown in FIG. 9. That is, the functions of the mixing engine are realized together by the selector and the robot related to the bypass media processing system. Here, the selector performs routing based on the energy of the energy upload and the corresponding routing decision.

[0079] Based on the AV conferencing realization method provided above, an embodiment of the present application further provides an AV conferencing realization apparatus. As shown in FIG. 10, the apparatus 1000 includes a receiving unit 1001 and a transfer unit 1002. The receiving unit 1001 is used to receive the pulse stream request of the first interface machine in the first device group. The pulse stream request includes the first identification information of the media data stream. The first device group is any one of the plurality of device groups. There is a back source device for each device group among the plurality of device groups, and each device group includes a plurality of interface machines. Each of the plurality of interface machines is used to connect a conference access terminal so that the conference access terminal can access the conference room through the interface machine. The first interface machine is any one of the interface machines in the first device group. The transfer unit 1002 is used to transfer the media data stream corresponding to the first identification information to the first interface machine so that the first interface machine transfers the media data stream to the corresponding conference access terminal. The media data stream is obtained from the second interface machine in the second device group by the back source device in the first device group. The conference access terminal that generates the media data stream accesses the conference room through the second interface machine, and the second device group is any one of the plurality of device groups.

[0080] In one possible embodiment, the transfer unit 1002 is used to search for the media data stream of the corresponding second interface machine according to the first identification information, if the media data stream of the second interface machine is not found after searching, transfer the pull stream request to the second interface machine, the receiving unit 1001 is further used to receive the media data stream transferred according to the first identification information in the pull stream request by the second interface machine, the transfer unit 1002 is further used to transfer the media data stream to the first interface machine.

[0081] In one possible embodiment, the transfer unit 1002 further if the media data stream of the corresponding second interface machine is found after searching, is used to transfer the media data stream to the first interface machine.

[0082] In one possible embodiment, when the first device group and the second device group are in different local area networks, the transfer unit 1002 transfers the pull stream request to the second interface machine through a relay agent device, is used to receive the media data stream transferred from the second interface machine through the relay agent device.

[0083] In one possible embodiment, the back source device in the first device group is obtained by being specified according to a rule preset by the first interface machine, and the preset rule routes the same media data stream to the interface machine in the device group to which it belongs through the same back source device.

[0084] In one possible embodiment, the apparatus further includes a sending unit, a recording unit, and a returning unit. When a third interface device in the third device group requests to enter the conference room, the receiving unit 1001 is further used to receive a registration request sent from the third interface device, where the third device group is any one of the plurality of device groups, and the third interface device is any one interface device in the third device group. If the sending unit fails to locally search for and find the information of the conference room, it is used to send the registration request to the central server so that the central server records the second identification information of the in-group server in the third device group in the first conference registration list and returns registration success information to the in-group server in the third device group. The recording unit is used to record the third identification information of the third interface device in the second conference registration list. The returning unit is used to return the registration success information to the third interface device.

[0085] In one possible embodiment, when the information of the conference room is found by local search, the recording unit is further used to record the third identification information of the third interface device in the second conference registration list. The returning unit is further used to return the registration success information to the third interface device.

[0086] In one possible embodiment, when the state of a conference access terminal connected to a third interface device in the third device group changes, the receiving unit 1001 is further used to receive a notification message sent from the third interface device. The sending unit is used to send the notification message to the center server so that the center server can send the notification message to the in-group servers of each device group, and the in-group servers of each device group are used to send the notification message to the interface machines that have successfully registered.

[0087] In one possible embodiment, when the third interface machine in the third device group requests a pull of the participation list, the receiving unit 1001 is further used to receive the list pull request of the third interface machine. If the sending unit fails to search for the participation list locally, it is used to send the list pull request to the center server. The receiving unit 1001 is further used to receive the participation list sent from the center server. The sending unit is used to send the participation list to the third interface machine.

[0088] In one possible embodiment, when the sending unit searches for the participation list locally and finds it, and the expiration date of the participation list has not expired, the sending unit is used to return the participation list to the third interface machine.

[0089] In one possible embodiment, the device further includes a selection unit and a mixing unit. When the media data stream includes an audio data stream, the sending unit is used to send the multi-audio data stream obtained by the selector in the mixing engine to the mixer in the same device group. The multi-audio data stream is from the fourth interface machines in different device groups. The selection unit is used to select a target audio data stream from the multi-audio data stream by the mixer. The transmitting unit is used to send a winning notice to the target interface device, which is an interface device connected to a conference access terminal that generates the target audio data stream. The receiving unit 1001 is further used to receive the target audio data stream transmitted from the target interface device by the mixer. The mixing unit is used to mix the target audio data stream. The transfer unit 1002 is further used to transfer the mixed audio data stream to a selector corresponding to the fourth interface device, and the fourth interface device includes the target interface device. The transmitting unit is further used to send the mixed audio data stream to the fourth interface device by the selector corresponding to the fourth interface device so that the fourth interface device transfers the mixed audio data stream to other interface devices in the device group to which it belongs.

[0090] In one possible embodiment, the apparatus further includes a processing unit. The processing unit is used to pull the media data stream generated in the conference room by a robot terminal and process the media data stream.

[0091] Embodiments of the present application provide an electronic device for realizing an AV conference. The electronic device may be a terminal, and an example will be described by taking the terminal as a smartphone.

[0092] FIG. 11 shows a block diagram of a partial configuration of a smartphone related to the terminal provided by the embodiment of the present application. Referring to FIG. 11, the smartphone includes components such as an RF (Radio Frequency) circuit 1110, a memory 1120, an input unit 1130, a display unit 1140, a sensor 1150, an audio circuit 1160, a WiFi (wireless fidelity) module 1170, a processor 1180, and a power supply 1190. The input unit 1130 may include a touch panel 1131 and other input devices 1132, the display unit 1140 may include a display panel 1141, and the audio circuit 1160 may include a speaker 1161 and a microphone 1162. As can be understood by those skilled in the art, the configuration of the smartphone shown in FIG. 11 does not limit the smartphone, and it may include more or fewer components than shown in the drawing, or some components may be combined, or different components may be arranged.

[0093] The memory 1120 may be used to store software programs and modules, and the processor 1180 may execute various functional applications and data processing of the smartphone by executing the software programs and modules stored in the memory 1120. The memory 1120 mainly includes a storage program area and a storage data area. Here, the storage program area can store an operating system, at least one application program required for functions (such as an audio playback function, an image playback function, etc.), and the storage data area can store data created based on the use of the smartphone (such as audio data, a phone book, etc.). Note that the memory 1120 may include a high-speed random access memory and may also include a non-volatile memory, for example, at least one disk memory device, a flash memory device, or other volatile solid-state memory devices.

[0094] The processor 1180 is the control center of the smartphone. By using various interfaces and lines to connect the individual parts of the entire smartphone, running or executing the software programs and / or modules stored in the memory 1120, and calling the data stored in the memory 1120, it executes various functions and processes data of the smartphone and monitors the smartphone as a whole. Preferably, the processor 1180 may include one or more processing units. Preferably, the processor 1180 may integrate an application processor and a modem processor. Here, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. As can be understood, the above modem processor may not be integrated into the processor 1180.

[0095] In this embodiment, the steps executed by the terminal in the above embodiment may be realized based on the configuration shown in FIG. 2.

[0096] The electronic device may further include a server. The embodiments of the present application further provide a server. As shown in FIG. 12, FIG. 12 is a configuration diagram of the server 1200 provided by the embodiments of the present application. The server 1200 may generate relatively large differences due to differences in arrangement or performance, and may include one or more central processing units (CPUs) 1222 (for example, one or more processors), a memory 1232, and a storage medium 1230 (for example, one or more large-capacity storage devices) that stores one or more application programs 1242 or data 1244. Here, the memory 1232 and the storage medium 1230 may be transient storage or persistent storage. The program stored in the storage medium 1230 may include one or more modules (not shown), and each module may include a series of command operations for the server. Further, the central processor 1222 may be installed to communicate with the storage medium 1230, and execute a series of command operations in the storage medium 1230 on the server 1200.

[0097] Server 1200 may include one or more power supplies 1226, one or more wired and wireless network interfaces 1250, one or more input / output interfaces 1258, and / or one or more operating systems 1241 such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, FreeBSD TM, etc.

[0098] In this embodiment, the central processor 1222 in the server 1200 can execute the following steps. That is Receive a pull stream request from a first interface device in the first device group, where the pull stream request includes first identification information of a media data stream, the first device group is any one of a plurality of device groups, there is a back source device for each device group among the plurality of device groups, each device group includes a plurality of interface devices, each of the plurality of interface devices is used to connect a conference access terminal so that the conference access terminal can access the conference room through the interface device, and the first interface device is any one of the interface devices in the first device group.

[0099] By transferring the media data stream corresponding to the first identification information to the first interface device, the first interface device transfers the media data stream to the corresponding conference access terminal. The media data stream is obtained from a second interface device in a second device group by a back source device in the first device group, and the conference access terminal that generates the media data stream accesses the conference room through the second interface device, and the second device group is any one of the plurality of device groups.

[0100] According to one aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium is used for storing program code. When the program code is executed on an electronic device, the AV conference realization method described in the individual embodiments is caused to be executed by the electronic device.

[0101] According to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program includes computer commands, and the computer commands are stored in a computer-readable storage medium. A processor of an electronic device reads the computer commands from the computer-readable storage medium, and the processor executes the computer commands to cause the electronic device to execute the methods provided in various preferred embodiments of the above embodiments.

[0102] Terms such as "first", "second", "third", "fourth", etc. (if any) in the specification and the above drawings of the present application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is understood that the data used in this way is interchangeable under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than, for example, those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units explicitly listed, and may include those not explicitly listed or other inherent steps or units for these processes, methods, products or devices.

[0103] In some embodiments provided by this application, it is understood that the systems, devices, and methods described may be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the partitioning of the said units is only a partitioning of logical functions. When actually implemented, it may have other partitioning methods. For example, a plurality of units or components may be combined or integrated into another system, or a certain feature may be ignored or not executed. On the other hand, the mutual coupling, direct coupling, or communication connection shown or discussed may be realized by the indirect coupling or communication connection of some interfaces, devices, or units, and may be electrical, mechanical, or in other forms.

[0104] The units described as the separation members may or may not be physically separated. The components as unit representations may or may not be physical units, that is, they may be located in one place or distributed among a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0105] Also, in individual embodiments of this application, individual functional units may be integrated into one processing unit, individual units may exist independently physically, or two or more units may be integrated into one unit. The above integrated units may be realized by adopting the form of hardware or the form of software functional units.

[0106] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. According to such an understanding, essentially the technical solution of this application, or the part that contributes to the prior art or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several commands to cause all or part of the steps of the method described in the individual embodiments of this application to be executed on a computer device (which may be a personal computer, a server, or a network device, etc.). The storage medium includes various media that can store program codes, such as a USB flash memory, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0107] As described above, the above embodiments are only used to explain the technical solution of this application and do not limit it. Even if this application is described in detail with reference to the above embodiments, those skilled in the art can modify the technical solutions described in the above embodiments or perform equivalent replacements on some of the technical features. It is understood that these modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of this application.

Description of Reference Signs

[0108] 301 Media Transmission Network 302 Room Management Subsystem 303 Mixing Engine 304 Bypass Media Processing System 305 Public Switched Telephone Network

Claims

1. A method for realizing an AV conference, comprising: a step in which a back source device in a first device group receives a pulse stream request from a first interface device in the first device group, wherein the pulse stream request includes first identification information of a media data stream, the first device group is any one of a plurality of device groups, each of the plurality of device groups has a back source device, each device group includes a plurality of interface devices, each of the plurality of interface devices is used to connect a conference access terminal so that the conference access terminal can access a conference room through the interface device, and the first interface device is any one of the interface devices in the first device group; a step in which the back source device in the first device group transfers the media data stream corresponding to the first identification information to the first interface device so that the first interface device transfers the media data stream to a corresponding conference access terminal, wherein the media data stream is obtained by the back source device in the first device group from a second interface device in a second device group, the conference access terminal that generates the media data stream accesses the conference room through the second interface device, and the second device group is any one of the plurality of device groups; The step in which the back source device in the first device group transfers the media data stream corresponding to the first identification information to the first interface device includes: a step of searching for a media data stream of the corresponding second interface device according to the first identification information; if the media data stream of the second interface device cannot be found after searching, a step of transferring the pulse stream request to the second interface device; a step of receiving the media data stream transferred according to the first identification information in the pulse stream request by the second interface device and transferring the media data stream to the first interface device. An AV conference realization method. Claim 2 The method according to claim 1, further comprising the step of transferring the media data stream to the first interface device when the media data stream corresponding to the second interface device is found by searching. Claim 3 When the first device group and the second device group are in different local area networks, the step of transferring the pull stream request to the second interface device includes the step of transferring the pull stream request to the second interface device via a relay agent device, and the step of receiving the media data stream transferred in response to the first identification information in the pull stream request by the second interface device includes the step of receiving the media data stream transferred from the second interface device via the relay agent device. The method according to claim 1. Claim 4 The backsource device in the first device group is obtained by being specified according to a rule preset by the first interface device, and the preset rule is such that the same media data stream is routed to the interface device in the device group to which it belongs via the same backsource device. The method according to claim 1. Claim 5 When a third interface device in a third device group requests to enter the conference room, the step of the in-group server in the third device group receiving a registration request transmitted from the third interface device, wherein the third device group is any one of the plurality of device groups, and the third interface device is any one interface device in the third device group, When the information of the conference room is not found by local search, the step of the in-group server in the third device group sending the registration request to the center server so that the center server records the second identification information of the in-group server in the third device group in a first conference registration list and returns registration success information to the in-group server in the third device group. The step in which the in-group server in the third device group records the third identification information of the third interface device in the second meeting registration list and returns the registration success information to the third interface device is further included in the method according to claim 1.

6. When the information of the meeting room is found by searching locally, the method according to claim 5 further includes the step in which the in-group server in the third device group records the third identification information of the third interface device in the second meeting registration list and returns the registration success information to the third interface device.

7. When the state of the meeting access terminal connected to the third interface device in the third device group is changed, the step in which the in-group server in the third device group receives the notification message sent from the third interface device The step in which the in-group server in the third device group sends the notification message to the center server so that the center server sends the notification message to the in-group servers of each device group, and the in-group server of each device group is used to send the notification message to the interface device with successful registration is further included in the method according to claim 5.

8. When the third interface device in the third device group requests a pull of the participation list, the step in which the in-group server in the third device group receives the list pull request of the third interface device When the participation list is not found by searching locally, the step in which the in-group server in the third device group sends the list pull request to the center server The method according to claim 5 further includes the step in which the in-group server in the third device group receives the participation list sent from the center server and sends the participation list to the third interface device.

9. The method according to claim 8 further includes the step in which the in-group server in the third device group searches locally for the participation list and, if found and the expiration date of the participation list has not expired, returns the participation list to the third interface device.

10. When the media data stream includes an audio data stream, a step of transmitting the acquired multi-audio data stream to a mixer in the same device group via a selector in the mixing engine, wherein each of the multi-audio data streams is from a fourth interface machine in a different device group, the step; A step of selecting a target audio data stream from among the multi-audio data streams by the mixer and transmitting a winning notice to the target interface machine, wherein the target interface machine is an interface machine connected to a conference access terminal that generates the target audio data stream, the step; Receiving the target audio data stream transmitted by the target interface machine via the mixer and mixing the target audio data stream, the step; A step of transferring the mixed audio data stream to a selector corresponding to the fourth interface machine, wherein the fourth interface machine includes the target interface machine, the step; A step of transmitting the mixed audio data stream to the fourth interface machine by a selector corresponding to the fourth interface machine so that the fourth interface machine transfers the mixed audio data stream to other interface machines in the device group to which it belongs, the method according to claim 1 including the step.

11. The method according to claim 1, further including a step of pulling a media data stream generated in the conference room by a robot terminal and processing the media data stream.

12. An AV conference realization device, A receiving unit used to receive a pulse stream request of a first interface machine in a first device group, wherein the pulse stream request includes first identification information of a media data stream, the first device group is any one of a plurality of device groups, each of the plurality of device groups has a back source device, each device group includes a plurality of interface machines, each of the plurality of interface machines is used to connect a conference access terminal so that the conference access terminal can access a conference room through the interface machine, and the first interface machine is any one of the first device group, the receiving unit, A transfer unit used to transfer the media data stream corresponding to the first identification information to the first interface machine so that the first interface machine transfers the media data stream to a corresponding conference access terminal, wherein the media data stream is obtained by a back source device in the first device group from a second interface machine in a second device group, the conference access terminal that generates the media data stream accesses the conference room through the second interface machine, and the second device group is any one of the plurality of device groups, the transfer unit, and The transfer unit is further used to search for the media data stream of the corresponding second interface machine according to the first identification information, and if the media data stream of the second interface machine is not found after the search, the transfer unit is further used to transfer the pulse stream request to the second interface machine, The receiving unit is further used to receive the media data stream transferred according to the first identification information in the pulse stream request by the second interface machine, the AV conference realization device.

13. An AV conference system, The AV conference system includes a data transmission network and a room management subsystem, The data transmission network is used to transmit the media data stream generated in the conference room. The data transmission network includes a plurality of device groups, and each device group includes a plurality of interface devices. Each of the plurality of interface devices is used to connect a conference access terminal so that the conference access terminal can access the conference room through the interface device. There is a back source device in each device group among the plurality of device groups. The back source device in the first device group receives the pull stream request of the first interface device in the first device group. The pull stream request includes the first identification information of the media data stream. It is used to transfer the media data stream corresponding to the first identification information to the first interface device. The media data stream is obtained by the back source device in the first device group from the second interface device in the second device group. The conference access terminal that generates the media data stream accesses the conference room through the second interface device. The first device group and the second device group are each any one of the plurality of device groups. The first interface device is any one of the interface devices in the first device group. The first interface device is used to transfer the media data stream to the corresponding conference access terminal. The room management subsystem includes the plurality of device groups and a center server. Each device group includes an in-group server. The in-group server is used to assist the center server in managing the conference room. Transferring the media data stream corresponding to the first identification information to the first interface device includes: Searching for the media data stream of the corresponding second interface device according to the first identification information; If the media data stream of the second interface device is not found after searching, transferring the pull stream request to the second interface device. Receiving the media data stream transferred according to the first identification information in the pulse stream request by the second interface device, and transferring the media data stream to the first interface device, an AV conferencing system including the above.

14. An electronic device for realizing an AV conference, The electronic device includes a processor and a memory, The memory stores program code and is used to transmit the program code to the processor, The processor is used to execute the method according to any one of claims 1 to 11 according to the commands in the program code, an electronic device for realizing an AV conference.

15. A computer program, When the computer program is executed, it causes an electronic device to execute the method according to any one of claims 1 to 11, a computer program.

Images