Audio and video conference method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an audio and video conference method and device, electronic equipment and a storage medium, belongs to the technical field of communication core networks, and can improve the convenience of participating in an audio and video conference. Comprising the following steps: receiving an incoming call signaling of a participant calling an intermediate number of an audio and video conference through a signaling server, sending the incoming call signaling to a communication media server, establishing a communication connection with a conference signaling server through the communication media server, and sending communication information to the conference media server by using the conference signaling server; according to the communication information, a monitoring address is reserved for the audio and video through the conference media server, and the monitoring address is synchronized to the communication media server; and under the condition that the intermediate number is connected, audio and video are mixed through the conference media server to obtain a single-path audio and video stream, and the single-path audio and video stream is pulled through the communication media server according to the monitoring address and is sent to the participant, so that the audio and video conference is carried out based on the single-path audio and video stream.

Description

Technical Field

[0001] The present application relates to the technical field of communication core networks, and in particular to a method, device, electronic device and storage medium for audio and video conferencing. Background Art

[0002] Currently, video conferencing based on Internet Protocol (IP) Multimedia Subsystem (IMS) network is very common.

[0003] In the related art, the methods of conducting video conferences through the IMS network mostly require conference participants to download the corresponding software of the conference or access the corresponding webpage before joining the conference, resulting in the problem of low convenience of participating in the conference. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for audio and video conferencing, so as to solve the problem of low convenience in participating in audio and video conferencing.

[0005] To solve the above technical problems, the embodiments of the present application are implemented as follows:

[0006] In the first aspect, an embodiment of the present application provides a method for audio and video conferencing, comprising: receiving, through a signaling server, an incoming signaling from a participant of an audio and video conference calling an intermediate number, and sending it to a communication media server; creating a communication connection with a conference signaling server through the communication media server, and using the conference signaling server to send communication information to the conference media server, the communication information including: a message of the participant joining the conference, and a media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, reserving a listening address for the audio and video through the conference media server, and synchronizing it to the communication media server, the listening address being used to push and pull the audio and video streams of the audio and video; when the intermediate number is connected, mixing the audio and video through the conference media server to obtain a single-channel audio and video stream, and according to the listening address, pulling the single-channel audio and video stream through the communication media server and sending it to the participant, so as to conduct the audio and video conference based on the single-channel audio and video stream.

[0007] In the second aspect, an embodiment of the present application provides a device for audio and video conferencing, including: a receiving device, used to receive incoming signaling from a participant of the audio and video conference calling an intermediate number through a signaling server, and send it to a communication media server; a creating device, used to create a communication connection with a conference signaling server through the communication media server, and use the conference signaling server to send communication information to the conference media server, the communication information including: a message of the participant joining the conference, and a media address reserved by the communication media server for the audio and video of the incoming signaling; a monitoring device, used to reserve a monitoring address for the audio and video through the conference media server according to the communication information, and synchronize to the communication media server, the monitoring address is used to push and pull the audio and video streams of the audio and video; a processing device, used to mix the audio and video through the conference media server when the intermediate number is connected, to obtain a single-channel audio and video stream, and according to the monitoring address, pull the single-channel audio and video stream through the communication media server and send it to the participant, so as to conduct the audio and video conference based on the single-channel audio and video stream.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-mentioned audio and video conferencing method.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program can be executed by a processor to implement the above-mentioned audio and video conferencing method.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the above-mentioned audio and video conferencing method.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the above-mentioned audio and video conferencing method.

[0012] By adopting the technical solution of the embodiment of the present application, the incoming signaling of the participant of the audio and video conference calling the middle number is received through the signaling server and sent to the communication media server, the communication connection with the conference signaling server is established through the communication media server, and the communication information is sent to the conference media server by using the conference signaling server, and the communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, the conference media server reserves a listening address for the audio and video and synchronizes it to the communication media server, the listening address is used to pull and push the audio and video streams of the audio and video, and when the middle number is connected, the conference media server mixes the audio and video to obtain a single-channel audio and video stream, and according to the listening address, the communication media server pulls the single-channel audio and video stream and sends it to the participant, so as to conduct an audio and video conference based on the single-channel audio and video stream. It can be seen that the participants of the meeting can connect to the audio and video conference by dialing the middle number. By reserving the media address and monitoring address for the audio and video corresponding to the participants, the audio and video streams corresponding to the audio and video can be pushed and pulled, allowing the participants to join the audio and video conference. The participants can enter the audio and video conference without downloading the audio and video conference software, thus solving the problem of low convenience in participating in audio and video conferences. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 It is a structural schematic diagram of an intermediate number platform according to an embodiment of the present application;

[0015] Figure 2 is a schematic flow chart of a method for audio and video conferencing according to an embodiment of the present application;

[0016] Figure 3 is a schematic diagram of media format conversion during a voice call according to an embodiment of the present application;

[0017] Figure 4 is a schematic diagram of media format conversion during an incoming video call according to an embodiment of the present application;

[0018] Figure 5 is a schematic diagram of a process of processing audio and video streams by a conference media server according to an embodiment of the present application;

[0019] Figure 6is a schematic swimlane diagram of a method for audio and video conferencing according to an embodiment of the present application;

[0020] Figure 7 is a schematic swimlane diagram of ending an audio or video conference according to an embodiment of the present application;

[0021] Figure 8 is a schematic flow chart of a method for audio and video conferencing according to another embodiment of the present application;

[0022] Fig. 9 is a schematic block diagram of an audio and video conferencing device according to an embodiment of the present application;

[0023] Fig.10 It is a schematic diagram of the hardware structure of an audio and video conferencing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application provide a method, device, electronic device and storage medium for audio and video conferencing, which are used to solve the problem of low convenience in participating in audio and video conferencing.

[0025] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0026] The audio and video conferencing method provided in the embodiment of the present application can be executed by an electronic device, or by software installed in the electronic device. Specifically, the electronic device can be a terminal device or a server device. Among them, the terminal device can include a smart phone, a laptop computer, a smart wearable device, a vehicle terminal, etc., and the server device can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0027] Figure 1 Schematic diagram of the structure of an intermediate number platform according to an embodiment of the present application. Figure 1As shown in the figure, the architecture of the intermediate number platform involves the information technology (IT) domain and the communication technology (CT) domain. Among them, in terms of architecture deployment, the deployment of the IT domain includes: conference service front-end, conference signaling service, and conference media service, and the deployment of the CT domain includes: signaling service and communication media service. The IT domain and the CT domain are connected through communication, using the interoperable and low-coupling architecture model of information and communications technology (ICT), so that each service can be deployed and expanded independently, which can maintain the stability of existing communication services and integrate new functions.

[0028] Among them, the CT domain is deployed in the IMS domain. It should be understood that the IMS domain also includes: Open Service Architecture (OSA) Service Capability Server (SCS), Open Service Architecture Application Server (OSA.AS), Home Subscriber Server (HSS), Registration Server (ServiceCall Session Control Function, S-CSCF), Multimedia Resource Function Controller (MRFC), Multimedia Resource Processor (MRFP), Cloud Service Engine (CSE). Specifically, a part of the media functions such as mixed stream processing are extracted from the communication media server. Since the communication media server is in the IMS communication network and the conference media server is outside the IMS communication network, the mixed stream processing part of the media function is extracted from the IMS communication network. The modularization and independence of the IMS communication network function can be achieved, which helps to improve the flexibility and scalability of the corresponding entire system, and facilitates the individual optimization, upgrade and maintenance of specific media functions without affecting the stable operation of the entire communication media server. At the same time, extracting the media function of mixed stream processing from the communication network can reduce the load and complexity of each server in the communication network and improve its overall performance and stability. In addition, since the conference media server is outside the communication network, it is easier to integrate and interact with external related systems or services.

[0029] In the following, in conjunction with the accompanying drawings, a method for audio and video conferencing provided by an embodiment of the present application is described in detail through specific embodiments and application scenarios.

[0030] Figure 2 A schematic flow chart of an audio and video conferencing method provided by an embodiment of the present invention is shown, and the method comprises the following steps:

[0031] S202, receiving an incoming signaling from a participant of an audio or video conference calling an intermediate number through a signaling server, and sending the signaling to a communication media server.

[0032] The signaling server (Application Server, AS) is similar to the role of the back-to-back user agent (Back-to-Back User Agent, B2BUA). In the communication network, the Session Initiation Protocol (Session Initiation Protocol, SIP) is used to implement the session. Specifically, the SIP protocol can be used to conduct signaling negotiation with the participants' terminals, and the signaling server is also used to transmit the data corresponding to the participants' audio and video to the communication media server.

[0033] The middle number is a virtual number provided for privacy protection or temporary communication. The audio and video conference room will generate a virtual number for entering the audio and video conference, namely the middle number. The signaling generated by the user dialing the middle number is the incoming signaling.

[0034] The communication media server (MS) is used to receive the data packets corresponding to the real-time audio and video of the participants sent by the signaling server, and can forward, record and save the data in the audio and video. In addition, the communication media server is also used to pull the mixed single-channel audio and video stream from the conference media server and push it to the participant's terminal through the core network, so that the participants can have real-time video conversations online.

[0035] When a participant applies to join an audio or video conference through a voice or video call intermediate number, call signaling is generated. The call signaling is routed to the signaling server through the core network. The signaling server receives the call signaling and synchronizes it to the communication media server based on the call signaling, so that the media information of the audio and video corresponding to the incoming signaling passes through the communication media server, where the media information includes: data corresponding to the participant's audio and video.

[0036] S204, establishing a communication connection with the conference signaling server through the communication media server, and using the conference signaling server to send communication information to the conference media server.

[0037] The communication information includes: the message that the participant joins the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling.

[0038] The conference signaling server (Central Application Server, CAS) is used to coordinate and manage audio and video conferences, such as allocating conference rooms, corresponding conference numbers, and synchronizing media, room numbers, and media addresses for pulling and pushing streams.

[0039] The communication connection includes a connection method of the full-duplex communication protocol websocket, that is, a websocket connection.

[0040] Conference Media Server (CMS) is used to aggregate, control, transcode, mix and synthesize the collected media resources, and supports various streaming media protocols.

[0041] The reserved media address specifically includes: the media resource applied for by the communication media server, which generates a media address for receiving the audio and video generated corresponding to the incoming signaling, and the audio and video include information such as video data and audio and video streams.

[0042] By creating a communication connection between the communication media server and the conference signaling server, the IT domain and the CT domain can be directly connected. The conference signaling server using the communication connection transmits the communication information to the conference media server. Since the communication information includes the message of the participant joining the conference and the media address reserved by the communication media server for the audio and video of the incoming signaling, the conference signaling server can carry the communication information and synchronize it to the conference media server so that the conference media server can determine the information of the participant joining the video conference and determine the media address of the audio and video generated by the participant in the communication media server.

[0043] S206, according to the communication information, reserve a monitoring address for the audio and video through the conference media server, and synchronize it to the communication media server.

[0044] The monitoring address is used to push and pull the audio and video streams.

[0045] Streaming refers to the process of transmitting packaged content to the server. Specifically, the data corresponding to the audio and video is encapsulated into streaming data using a streaming protocol. Commonly used streaming protocols include the Real-Time Messaging Protocol (RTMP).

[0046] Stream pulling refers to the process in which the communication media server and the conference media server establish a connection and pull data according to the RTMP protocol. Specifically, the communication media server pulls a single-channel audio and video stream from the conference media server.

[0047] Optionally, the listening address includes a streaming address for streaming and a streaming address for streaming.

[0048] According to the communication information, the audio and video corresponding to the participant can be determined, and the conference media server reserves a monitoring address for the audio and video and synchronizes it to the communication media server, so that the conference media server and the communication media server can push and pull the audio and video through the monitoring address. The monitoring address can be a network resource for pushing and pulling the audio and video streams of the audio and video.

[0049] S208, when the intermediate number is connected, the audio and video are mixed through the conference media server to obtain a single-channel audio and video stream. According to the monitoring address, the single-channel audio and video stream is pulled through the communication media server and sent to the participants to conduct an audio and video conference based on the single-channel audio and video stream.

[0050] Audio and video streaming refers to the audio and video data corresponding to the audio and video transmitted in the form of a stream in the network. This transmission method allows the audio and video data to be downloaded and played at the same time, thus achieving real-time or near real-time audio and video playback.

[0051] After S204 reserves the media address for the audio and video corresponding to the participant, and S206 reserves the monitoring address for the audio and video corresponding to the participant, the participant can dial the middle number to dial the audio and video conference and enter the audio and video conference. When the middle number is connected, the audio and video are mixed, that is, the audio and video streams corresponding to the audio and video generated by the participant are mixed to obtain a mixed single-channel audio and video stream.

[0052] It should be noted that there are one or more participants, and there can be one or more audio and video generated by the participants, that is, there are one or more audio and video streams corresponding to the audio and video generated by the participants, and the conference media server mixes the audio and video streams corresponding to the audio and video generated by the participants to obtain one mixed audio and video stream. It can be seen that the conference media server can mix multiple audio and video streams into a single audio and video stream. Among them, the first participant enters the audio and video conference through the intermediate number, and some initial resource data of the intermediate number platform will be initialized. Participants who enter the audio and video conference later do not need to initialize the data, and the remaining process is the same as the first participant, and no specific limitation is made here.

[0053] The conference media server can determine the monitoring address of the mixed single-channel audio and video stream, and the communication media server pulls the mixed single-channel audio and video stream. The communication media server sends the pulled single-channel audio and video stream to the participants through the core network, and the participants conduct an audio and video conference based on the single-channel audio and video stream.

[0054] By adopting the technical solution of the embodiment of the present application, the incoming signaling of the participant of the audio and video conference calling the middle number is received through the signaling server and sent to the communication media server, the communication connection with the conference signaling server is established through the communication media server, and the communication information is sent to the conference media server by using the conference signaling server, and the communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, the conference media server reserves a listening address for the audio and video and synchronizes it to the communication media server, the listening address is used to pull and push the audio and video streams of the audio and video, and when the middle number is connected, the conference media server mixes the audio and video to obtain a single-channel audio and video stream, and according to the listening address, the communication media server pulls the single-channel audio and video stream and sends it to the participant, so as to conduct an audio and video conference based on the single-channel audio and video stream. It can be seen that the participants of the meeting can connect to the audio and video conference by dialing the middle number. By reserving the media address and monitoring address for the audio and video corresponding to the participants, the audio and video streams corresponding to the audio and video can be pushed and pulled, allowing the participants to join the audio and video conference. The participants can enter the audio and video conference without downloading the audio and video conference software, thus solving the problem of low convenience in participating in audio and video conferences.

[0055] In one embodiment, after receiving an incoming signaling from a participant of an audio or video conference calling an intermediate number through a signaling server and sending it to a communication media server (ie, S102), the following steps A1-A2 may be performed:

[0056] Step A1: Notify the communication media server to reserve a media address for the audio and video of the incoming signaling.

[0057] When a participant generates a call signaling through a voice or video call intermediate number, the signaling server receives the call signaling and sends it to the communication media server. It can then request the communication media server to apply for media resources for the incoming signaling. After receiving the request, the communication media server reserves a media address for the audio and video of the incoming signaling. Specifically, the communication media server pre-allocates a media address for receiving the media information corresponding to the audio and video of the call. The media information specifically includes: the media encoding, payload type, IP, media format, port, supported audio, and other information corresponding to the audio and video.

[0058] Step A2: Return the media address to the signaling server, and determine the media format of the participant's audio and video through the signaling server.

[0059] According to the media address reserved by the communication media server in step A1, the media address is returned to the signaling server, and the signaling server unifies the media format of the participants' audio and video to obtain the media format of the participants' audio and video, wherein the address used by the audio and video in the unified media format is the media address.

[0060] In this embodiment, the communication media server reserves the media address for the audio and video and returns it to the signaling server. The media address is received through the signaling server, and the media format of the participants' audio and video is uniformly processed to determine the media format of the participants' audio and video. The audio and video in the uniform media format can flow into the communication media server through the signaling server, and the intermediate number platform and the participants' terminals can communicate using a uniform media format to the greatest extent, thereby reducing the transcoding overhead and the problem of media information synchronization.

[0061] In one embodiment, the signaling server determines the media format of the participant's audio and video (i.e., step A2), and the following step A21 may be performed:

[0062] Step A21: When the incoming signaling is a voice call, the signaling server and the participant negotiate the media format of the audio and video and the video, and change the media format of the audio and video to the negotiated media format.

[0063] Media formats include audio, video, images and other types, and each type has multiple formats to choose from.

[0064] When the incoming signaling is a voice call, the media format of the audio is formed. The signaling server and the participants negotiate the media format of the generated audio and video to generate a unified media format. The unified media format is the same media format as the participants and the signaling server. All participants use this media format for audio and video conferences. If there are multiple participants, each participant negotiates with the signaling server and processes the media format of the audio and video generated by the participants into the unified media format of the signaling server.

[0065] Video negotiation means that since there is no video camera function in the voice call, it is necessary to negotiate with the participants through video pulling to determine whether to turn on the camera function of the participant's terminal in order to conduct an audio and video conference.

[0066] Specifically, by determining that the incoming signaling is a voice incoming signaling, the signaling server negotiates with the participant to convert the media format of the audio and video into a unified media format in the signaling server, wherein the address of the converted media format is a media address reserved by the communication media server, and video negotiation is performed with the participant to enable the participant to turn on the video function.

[0067] Optionally, when the incoming signaling is a video call, the signaling server and the participants negotiate the media format of the generated audio and video to generate a unified media format. The unified media format is a media format consistent with the participants and the signaling server, and all participants use this media format for audio and video conferences. For example, when there are multiple participants, each participant negotiates with the signaling server and processes the media format of the audio and video generated by the participants into the unified media format of the signaling server.

[0068] Specifically, by determining that the incoming signaling is a video incoming signaling, the signaling server converts the media format of the audio and video into a unified media format in the signaling server. Since the camera in the terminal of the participant of the video incoming signaling is turned on, the Service Delivery Platform (SDP) in the video incoming signaling already carries the video offer, and there is no need to add an additional process to start the video negotiation process. The address for converting the media format is a reserved media address.

[0069] In this embodiment, based on the incoming signaling, the signaling server and the participants perform half-end negotiation. When it is a voice call, video negotiation is required with the participant's terminal. If it is a video call, video negotiation is not required. The signaling server and the participants negotiate the media format, and the media format of the audio and video is converted into a unified media format of the signaling server, which can reduce the transcoding overhead. The address used for the negotiation adopts the media address reserved by the communication media server, which facilitates the flow of audio and video streams corresponding to the audio and video into the communication media server, thereby improving the processing efficiency of audio and video.

[0070] In one embodiment, before the communication media server establishes a communication connection with the conference signaling server (ie, S106), the following steps B1-B2 may be performed:

[0071] Step B1: The audio and video information after the media format of the audio and video is determined is synchronized to the communication media server and stored in the media address of the communication media server.

[0072] The audio and video information after determining the media format of the audio and video, wherein the audio and video information can be media information. Since the resource address corresponding to the audio and video information is the media address reserved by the communication media server, when the signaling server synchronizes the audio and video information to the communication media server, the audio and video information can be stored in the media address of the communication media server, so that the communication media server can perform operations such as transcoding, media forwarding, and recording on the audio and video data of the two parties to the call.

[0073] Step B2: Send a start message for connecting the audio and video conference to the participants through the signaling server, and send the start message to the communication media server.

[0074] The start message may include a ringing message, etc. As an example, Figure 3As shown, it is a schematic diagram of media format conversion during voice incoming call. Specifically, the participant sends a first request to the signaling server and receives the first received signaling. The first request is used to request the signaling server to receive the incoming signaling sent by the participant. The first reception is used to notify the participant that the signaling server has received the incoming signaling. The interaction can be carried out through codes such as: request (offer: audio data) INVITE (offer: audio sendrecv), request (receive: audio data) 183 (answer: audio sendrecv). The signaling server receives the first request and notifies the communication media server to create an audio and video conference call for the audio and video corresponding to the incoming signaling, as well as the reserved audio and video media address. The communication media server returns the reserved audio and video media address to the signaling server. The participant and the signaling server conduct signaling negotiation, wherein the first temporary response confirmation is used to indicate the receipt of the information sent by the signaling server, and the response to the first temporary response is used to indicate the confirmation of the final response of the signaling server to the received message. The specific codes include: provisional response confirmation (Provisional Response ACKnowledgement, PRACK), response to the first temporary response 200-PRACK. The incoming signaling carries the media information of the participants. The signaling server receives the media information of the participants' audio and video, and synchronizes the media information and media address to the communication media server. The first resource update includes converting the media format of the audio and video into media information formed by a unified media format in the signaling server, and updating it in the reserved media address. The response to the first resource update includes the participants receiving the updated media information and media address. The specific codes for interaction between the participants and the signaling server include: update UPDATE, 200-UPDATE. By judging that the incoming signaling is a voice incoming signaling, there is no camera function in the voice incoming call. Therefore, the signaling server and the participant start a video negotiation, and turn on the camera function of the participant terminal by pulling the video negotiation to conduct an audio and video conference. For example, the second resource update includes video negotiation with the participant through the signaling media server, and the purpose is to turn on the camera function of the participant terminal. The response to the second resource update includes turning on the camera function of the participant terminal. The specific code may include: UPDATE (offer: audio sendrecv, video sendrecv), 200-UPDATE (offer: audio sendrecv, video sendrecv).The signaling server synchronizes the negotiated audio and video as well as the media information and media address converted into a unified media format to the communication media server. The call can start, and the start message of the call is sent to the communication media server and the participant. The first ringing message is used to indicate that a ringing notification is sent to the participant to indicate that the call can start. The response to the first request is used to indicate that the participant's initial request has been received. The first final confirmation is used to indicate that the data has been received correctly. Among them, some interaction codes between the participant and the signaling server can be represented as: 180, 200-INVITE, Acknowledge character (ACK).

[0075] As another example, Figure 4As shown, it is a schematic diagram of media format conversion during video incoming call. Specifically, the participant sends a second request to the signaling server and receives the second received signaling. The second request is used to request the signaling server to receive the incoming signaling sent by the participant. The second reception is used to notify the participant that the signaling server has received the incoming signaling. The interaction can be carried out through codes such as INVITE (offer: audio sendrecv), 183 (answer: audio sendrecv). The signaling server receives the second request and notifies the communication media server to create an audio and video conference call for the audio and video corresponding to the incoming signaling, and to reserve the media address for receiving the audio and video. The communication media server returns the reserved audio and video media address to the signaling server. The participant and the signaling server conduct signaling negotiation, wherein the second temporary response confirmation is used to indicate the receipt of the information sent by the signaling server, and the response to the second temporary response is used to indicate the confirmation of the final response of the signaling server to the received message. The specific codes include: PRACK, 200-PRACK. The incoming signaling carries the media information of the participant. The signaling server receives the media information of the participant's audio and video and synchronizes the media information to the communication media server. Among them, the third resource update includes converting the media format of audio and video into the media information formed by the unified media format in the signaling server, and updating it in the reserved media address. The response to the third resource update includes the participant receiving the updated media information and media address and other information. The specific codes for the interaction between the participant and the signaling server include: UPDATE, 200-UPDATE. By judging that the incoming signaling is a video incoming signaling, there is no need to add additional processes to pull up the video negotiation process. Finally, the signaling server synchronizes the media information converted into a unified media format to the communication media server, and the call can start. The call start message is sent to the communication media server and the participant. The second ringing message is used to indicate that a ringing notification is sent to the participant to indicate that the call can start. The response to the second request is used to indicate that the participant’s initial request has been received. The second final confirmation is used to indicate that the data has been received correctly. Among them, some of the interaction codes between the participant and the signaling server can be represented as: 180, 200-INVITE, ACK.

[0076] In this embodiment, the audio and video information after the media format of the audio and video is determined is synchronized to the communication media server and stored in the media address of the communication media server. The stored audio and video can be processed later by the communication media server; at the same time, the signaling server sends a start message to the participants and the communication media server, which can inform the participants that they are accessing the video conference, and can notify the communication media server that network resources need to be reserved for the participants' audio and video through the conference media server.

[0077] In one embodiment, according to the communication information, a monitoring address is reserved for audio and video through the conference media server and synchronized to the communication media server (ie, S108), and the following steps C1-C3 may be specifically performed:

[0078] Step C1: according to the communication information sent by the conference signaling server, the RTMP streaming media server including the RTMP protocol is started through the conference media server.

[0079] In RTMP streaming media server, streaming media refers to a media form that transmits audio, video and multimedia files in a streaming manner in the network. The main function of the streaming media server is to transmit audio and video files to the client using a streaming protocol for users to watch online. It can also receive real-time audio and video streams through video capture and compression software, and then send them to the client using a streaming protocol. The streaming protocol includes the RTMP protocol.

[0080] Since the communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the participant to push the stream. After the conference signaling server sends the communication information to the conference media server, the conference media server receives the message of the participant joining the conference and determines the media address reserved by the communication media server. After that, the conference media server can start the RTMP streaming media server including the RTMP protocol.

[0081] Step C2: Based on the RTMP streaming media server, a monitoring address is reserved for audio and video.

[0082] According to the RTMP streaming media server, the conference media server can reserve a listening address for participants to pull and push streams through the RTMP protocol in the RTMP streaming media server. Specifically, the listening address includes the IP address and port required in the process of pulling and pushing streams. The listening address is similar to a link channel for pushing media data. Among them, the listening address includes the push stream address and the pull stream address. The push stream address can be the push stream address of multiple participants, but the pull stream address is the same pull stream address.

[0083] Step C3: The monitoring address is returned to the conference signaling server through the conference media server, and the monitoring address is synchronized to the communication media server through the conference signaling server.

[0084] The conference media server synchronizes the listening address to the conference signaling server and sends it to the communication media server, so that the communication media server and the conference media server can push and pull the audio and video streams through the listening address.

[0085] In this embodiment, based on the communication information sent by the conference signaling server, the conference media server can start the RTMP streaming media server, reserve a listening address for the audio and video streams generated by the participants, and synchronize it to the communication media server, which can facilitate the subsequent communication media server and conference media server to push and pull the audio and video streams.

[0086] In one embodiment, after the conference media server reserves a monitoring address for the audio and video and synchronizes it to the communication media server (ie, S108), the following steps D1-D2 may be performed:

[0087] Step D1: Use the communication media server to send a reservation message to the signaling server.

[0088] The reservation message includes a message of successfully reserving a media address and a monitoring address for audio and video.

[0089] The communication media server is used to notify the signaling server of the successful reservation of the media resources of the reserved network for the audio and video of the participant, wherein the media resources include a media address and a monitoring address.

[0090] Step D2: According to the reservation message, it is sent to the participants through the signaling server.

[0091] The signaling server synchronizes the reservation message to the participants. For example, the signaling server returns a 200OK status code to participant A, allowing the participant to call the intermediate number and be successfully connected.

[0092] In this embodiment, the communication media server sends a reservation message to the signaling server, communicates with the participants through the signaling server, and informs the participants that the call to the middle number is connected, so that the participants can confirm to join the audio and video conference and make an audio and video call.

[0093] In one embodiment, the conference media server performs mixing processing on the audio and video to obtain a single-channel audio and video stream (ie, S108), and the following step E may be performed:

[0094] Step E: Receive audio and video through the conference media server, and mix the audio and video streams corresponding to the audio and video.

[0095] The mixing process includes processing the audio and video streams corresponding to the multiple audio and video channels into a single audio and video stream.

[0096] The hybrid processing includes the conference media server processing the audio and video streams by means of a multipoint control unit (MCU), specifically decoding the audio and video streams corresponding to one or more received audio and video, re-laying out and mixing the decoded audio and video streams for video synthesis, then mixing the audio and video streams by mixing and resampling, and finally generating a single audio and video stream by encoding and media format encapsulation and synchronization processing. Among them, since the media format of audio and video is the same, mixing and resampling can be performed.

[0097] It should be noted that after the communication media server pulls the mixed single-channel audio and video stream, it is forwarded to the participants through the core network. In addition, the audio and video streams of multiple audio and video received by the conference media server are the audio and video streams pushed by the communication media server through the push stream address in the listening address. The single-channel audio and video stream pulled by the communication media server after mixing is the audio and video stream pulled through the pull stream address. Since it is a single-channel audio and video stream, the pull stream address is one.

[0098] As an example, Figure 5 As shown, it is a schematic diagram of the conference media server processing the audio and video stream. Participants include participant A, participant B and participant C; the communication media server cluster includes communication media server 1, communication media server 2 and communication media server 3, participant A corresponds to communication media server 1, participant B corresponds to communication media server 2, and participant C corresponds to communication media server 3. Each participant pushes audio stream and video stream to the communication media server, and the communication media server can also push audio stream and video stream to the participant at the same time. The single-channel audio and video stream pushed by the communication media server to the participant is a single-channel audio and video stream after pulling and synthesizing. Among them, after the communication media server receives the audio stream and video stream pushed by the participant, it pushes the participant's audio and video stream to the conference media server through the push stream address of the listening address in the RTMP protocol, and decodes the received audio and video stream in the conference media server to obtain the decoded audio and video stream, and synthesizes the audio and video stream by re-layout and mixing, and mixes the synthesized video stream by mixing and resampling, and synchronizes the mixed multi-channel audio and video streams and encodes and encapsulates them to generate a single-channel audio and video stream. The communication media server pulls the single-channel audio and video stream from the conference media server by listening to the stream pull address in the address.

[0099] In this embodiment, by performing mixing processing on multiple audio and video streams, such as sound mixing and video synthesis, and processing them into a single audio and video stream compatible with the core network, the transmission efficiency of audio and video can be improved and the user experience can be enhanced.

[0100] In one embodiment, before receiving the incoming signaling (i.e., S102) from a participant of the audio or video conference calling the intermediate number through the signaling server, the following steps F1-F3 may be performed:

[0101] Step F1: Acquire the audio and video conference established by the user through the conference service front end, and send a request to create an audio and video conference to the conference signaling server.

[0102] The conference service front end refers to the form of software or web page that is used to provide functions such as booking meetings and querying meeting information.

[0103] Step F2: Send the request to the conference media server through the conference signaling server, and use the conference media server to create a conference room.

[0104] The conference signaling server sends a request to create an audio and video conference to the conference media server. The conference media server creates a conference room and sets a scheduled time. Participants can join the audio and video conference only during the scheduled time.

[0105] Step F3: Use the conference signaling server to assign an intermediate number to the conference room, and send the intermediate number to the conference service front end for display.

[0106] The conference signaling server receives the conference room created for the participant in step F2, assigns a corresponding intermediate number to the conference room, and sends the intermediate number to the conference service front end for display. The participants of the video conference can join the audio and video conference by calling the intermediate number within the scheduled time.

[0107] It should be noted that the applied intermediate number can be reused, and there is no need to reallocate a new number or identifier every time an audio or video conference is organized. For example, the user who organizes the audio or video conference organizes the audio or video conference within the scheduled time. When he wants to organize the audio or video conference again at another time, he can directly modify the scheduled time and join the audio or video conference by dialing the intermediate number. This has better cost-effectiveness and operational convenience. Of course, if you want to reapply for the audio or video conference to obtain the corresponding intermediate number, you can also reapply, and there is no specific limitation here.

[0108] In this embodiment, the conference service front end requests the conference signaling server to create a conference room, the conference room is created through the conference media server, and the conference signaling server allocates an intermediate number and displays it on the conference service front end. Participants can directly enter the audio and video conference by dialing the intermediate number without downloading the corresponding software for the audio and video conference, etc., thereby saving the time and energy of users applying for audio and video conferences, improving the efficiency of convening audio and video conferences, and also improving the convenience and flexibility of participants joining audio and video conferences.

[0109] In one embodiment, the method may further perform the following steps G1-G3:

[0110] Step G1: When a participant exits the audio and video conference, a signaling server receives an exit message from the participant and sends the exit message to a communication media server.

[0111] Participants exiting the audio and video conference can hang up the middle number for the participants.

[0112] The signaling server can receive an exit message from a participant to exit the audio and video conference, wherein the exit message from the participant to exit the audio and video conference, for example, the signaling server receives a goodbye BYE message and sends the exit message to the communication media server so that the communication media server ends the participant's call.

[0113] Step G2: Stop pushing and pulling the audio and video of the participant through the communication media server, and send a notification of the participant exiting the conference to the conference signaling server.

[0114] The communication media server receives the exit message of the participant from the audio and video conference sent by the signaling server, stops the push stream processing of the participant's audio and video, and stops the pull stream processing from the conference media server. At the same time, it sends a notification of the participant's exit from the conference to the conference signaling server.

[0115] Step G3: Use the conference signaling server to send a notification of exiting the conference to the conference media server, and remove the participant from the audio and video conference through the conference media server.

[0116] The conference media server removes the participant from the audio or video conference and closes the RTMP connection allocated to the participant.

[0117] In this embodiment, when a participant actively exits the audio and video conference, the signaling server receives the participant's exit message from the audio and video conference and sends it to the communication media server. The communication media server stops pushing and pulling the audio and video streams corresponding to the participant's audio and video, and notifies the participant of the synchronous exit from the conference. When the participant actively exits the conference, the communication media server can immediately stop processing the participant's audio and video streams.

[0118] In one embodiment, the method may further perform the following steps H1-H3:

[0119] Step H1: When the audio and video conference ends, a conference end notification is sent to the communication media server through the conference signaling server, and the communication media server is notified to close the audio and video conference.

[0120] When the audio and video conference ends, that is, the time to create the conference expires or the user who applied to create the audio and video conference ends the conference, the conference signaling server sends a notification of the end of the conference to the communication media server, and the communication media server closes the audio and video conference room and stops the related RTMP connection.

[0121] Step H2: Use the communication media server to stop the push and pull processing of audio and video, and synchronize the notification of the end of the meeting to the signaling server.

[0122] In step H1, the communication media server receives the notification of the end of the meeting, closes the audio and video conference, stops pushing and pulling the audio and video streams corresponding to the participants, plays the reminder sound of the end of the meeting to all online participants of the video conference, and synchronizes the notification of the end of the meeting to the signaling server.

[0123] Step H3: Send a notification of the end of the conference to the participants via the signaling server.

[0124] After receiving the notification of the end of the conference, the signaling server sends BYE to all online participants to end the call.

[0125] In this embodiment, when the audio and video conference expires, the conference signaling server sends a notification of the end of the conference to the communication media server, and the communication media server closes the audio and video conference, stops the push and pull processing of the audio and video, synchronizes the notification of the end of the conference to the signaling server, and the signaling server sends the notification of the end of the conference to the participants. When the audio and video conference ends, the processing of the audio and video streams of the participants can be stopped immediately, and the participants are notified of the end of the audio and video conference.

[0126] In summary, a method for audio and video conferencing provided by this embodiment is described below through a specific scenario. In this embodiment, the audio and video conferencing is applied to the intermediate number platform. The participants of the audio and video conference are A and B. Through the interaction between participant A, the conference service front end, the conference signaling server, the conference media server, the signaling server, the communication media server and participant B, participant A and participant B can join the audio and video conference. Figure 6 The embodiment shown is described as follows. Figure 6 As shown, the following steps S6.1-S6.42 are included:

[0127] S6.1: The conference service front end sends a request to create an audio or video conference to the conference signaling server.

[0128] S6.2: The conference signaling server sends a request to the conference media server to create a room corresponding to the audio and video conference.

[0129] S6.3: The conference media server creates an audio and video conference room.

[0130] S6.4: The conference media server returns the room number of the audio and video conference to the conference signaling server.

[0131] S6.5: The conference signaling server allocates an intermediate number for the audio and video conference and associates it with the room number.

[0132] S6.6: The conference signaling server returns the audio and video conference intermediate number to the conference service front end and displays it.

[0133] S6.7: Participant A calls the middle number of the audio and video conference.

[0134] S6.8: The signaling server sends a message to the communication media server to create an audio or video conference call.

[0135] S6.9: The communication media server allocates media resources and IP ports to the signaling server.

[0136] S6.10: Participant A and the signaling server perform half-end negotiation.

[0137] S6.11: The signaling server and the communication media server synchronize the media data of the audio and video and the media address of participant A, etc.

[0138] S6.12: Participant A and the signaling server send a ringing message to connect to the intermediate number.

[0139] S6.13: The signaling server sends a message to the communication media server to start the call.

[0140] S6.14: The communication media server creates a websocket connection with the conference signaling server.

[0141] S6.15: Send a message to the conference signaling server that participant A joins the audio and video conference with room number middle, and carry the media address.

[0142] S6.16: The conference signaling server sends a message to the conference media server indicating that participant A has joined the corresponding audio and video conference room.

[0143] S6.17: The conference media server starts the RTMP streaming server and returns the RTMP streaming address and the streaming address of participant A.

[0144] S6.18: The conference media server returns the monitoring address to the conference signaling server.

[0145] S6.19: The conference signaling server returns the RTMP streaming address and streaming address to the communication media server.

[0146] S6.20: The communication media server sends a notification to the signaling server that participant A has joined the audio and video conference.

[0147] S6.21: Participant A connects to the audio and video conference, wherein media data can be transmitted between participant A and the communication conference server through the core network.

[0148] S6.22: The communication conference server pushes the audio and video stream of participant A to the conference media server via the RTMP protocol.

[0149] S6.23: The conference media server receives the audio and video streams of participant A and performs audio mixing and video synthesis using the MCU method.

[0150] S6.24: The communication media server pulls the synthesized single-channel audio and video stream from the conference media server through the RTMP protocol.

[0151] S6.25: Participant B sends a call signaling to the signaling server to call the intermediate number.

[0152] S6.26: The signaling server sends a message to the communication media server to create an audio or video conference call.

[0153] S6.27: The communication media server allocates media resources and IP ports to the signaling server.

[0154] S6.28: Participant B and the signaling server perform half-end negotiation.

[0155] S6.29: The signaling server and the communication media server synchronize the media information of the audio and video and the media address of participant B.

[0156] S6.30: Participant B and the signaling server send a ringing message to connect to the intermediate number.

[0157] S6.31: The signaling server sends a message to the communication media server to start the call.

[0158] S6.32: The communication media server creates a websocket connection with the conference signaling server.

[0159] S6.33: Send a message to the conference signaling server that participant A joins the audio and video conference with room number middle, and carry the media address.

[0160] S6.34: The conference signaling server sends a message to the conference media server indicating that participant A has joined the corresponding audio and video conference room.

[0161] S6.35: The conference media server starts the RTMP streaming server and returns the RTMP streaming address and the streaming address of participant B.

[0162] S6.36: The conference media server returns the monitoring address to the conference signaling server.

[0163] S6.37: The conference signaling server returns the RTMP streaming address and streaming address to the communication media server.

[0164] S6.38: The communication media server sends a notification to the signaling server that participant A has joined the audio and video conference.

[0165] S6.39: Participant B connects to the audio and video conference, wherein media data can be transmitted between participant B and the communication conference server through the core network.

[0166] S6.40: The communication conference server pushes the audio and video stream of participant B to the conference media server via the RTMP protocol.

[0167] S6.41: The conference media server receives the audio and video streams of participant B and performs audio mixing and video synthesis using the MCU method.

[0168] S6.42: The communication conference server pulls the synthesized single-channel audio and video stream from the conference media server through the RTMP protocol.

[0169] The specific process of S6.1-S6.42 has been described in detail in the above embodiment and will not be repeated here.

[0170] In this embodiment, through the interaction between participant A, conference service front end, conference signaling server, conference media server, signaling server, communication media server and participant B, the participant can join the audio and video conference by dialing the middle number, and join the audio and video conference by adding the middle number of the audio and video conference without changing other contents of the audio and video conference, thereby improving the convenience of the participant to join the audio and video conference.

[0171] like Figure 7 As shown, a swim lane diagram for ending an audio or video conference is shown, which may include the following steps S7.1-S7.14:

[0172] S7.1: When participant A exits the audio and video conference, participant A sends a hang-up notification to the signaling server.

[0173] S7.2: The signaling server sends a notification to the communication media server to stop the call of participant A.

[0174] S7.3: The communication media server sends a notification of participant A exiting the audio and video conference to the conference signaling server through the communication connection with the conference signaling server.

[0175] S7.4: When participant A exits the audio and video conference, the communication media server stops pushing the stream to participant A and stops pulling the stream from the conference media server.

[0176] S7.5: The conference signaling server sends a message to the conference media server indicating that participant A has exited the audio and video conference room.

[0177] S7.6: The conference media server removes participant A from the audio and video conference room and closes the RTMP connection allocated to participant A.

[0178] S7.7: The conference media server notifies the conference signaling server to confirm the deletion of participant A.

[0179] S7.8: When an audio or video conference times out or is actively closed by the initiator, the audio or video conference ends.

[0180] S7.9: The communication media server stops pushing and pulling streams for participant A, and plays the audio of the end of the conference to conference participants A and B.

[0181] It should be noted that there can be one or more participants in an audio or video conference, and there is no specific limitation.

[0182] S7.10: Notify the signaling server that the audio and video conference has ended.

[0183] S7.11: Send a notification of hang-up to the participant.

[0184] As shown in the figure, a hang-up notification is sent to participant B, and a hang-up notification is also sent to participant A at the same time.

[0185] S7.12: At the same time, the conference signaling server sends a message to the conference media server to close the room corresponding to the audio and video conference.

[0186] S7.13: The conference media server closes the room and releases the room space and related RTMP connections.

[0187] S7.14: The conference media server sends a notification confirming the closing to the conference signaling server.

[0188] The specific process of S7.1-S7.14 has been described in detail in the above embodiment and will not be repeated here.

[0189] In this embodiment, by communicating and connecting the server in the IT domain and the server in the CT domain, participants can actively exit the audio and video conference without changing the original audio and video conference information, because the audio and video conference can be ended due to timeout, or the initiator of the audio and video conference can also end the audio and video conference, thereby improving the flexibility of exiting the audio and video conference.

[0190] Figure 8 is a schematic flow chart of a method for audio and video conferencing according to another embodiment of the present application, such as Figure 8 As shown, the method comprises the following steps:

[0191] S801, the conference service front end sends a request for creating an audio or video conference to the conference signaling server, and sends the request to the conference media server through the conference signaling server.

[0192] S802, the conference media server creates a room number for the audio and video conference, and sends the room number to the conference signaling server. The conference signaling server allocates an intermediate number to the room number and displays the intermediate number on the conference service front end.

[0193] S803, the conference participant generates an incoming signaling by dialing the intermediate number, and sends it to the communication media server through the signaling server. The communication media server reserves a media address for the audio and video of the incoming signaling and returns the media address to the signaling server.

[0194] S804, the signaling server and the participant conduct half-end negotiation to change the media format of the audio and video to a media format supported by the communication media server.

[0195] In the case where the incoming signaling is a voice call, a video negotiation is performed between the signaling server and the participant, and the camera of the participant's terminal is turned on for audio and video conferencing.

[0196] S805, synchronizing the media information of the audio and video in a unified media format to the communication media server, and the signaling server sends a start message of the call start to the participant and the communication media server.

[0197] S806, the communication media server establishes a communication connection with the conference signaling server, and sends communication information to the conference signaling server.

[0198] The communication information includes: the message that the participant joins the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling.

[0199] S807, the conference signaling server sends the communication information to the conference media server, and the conference media server starts the RTMP streaming media server and generates a listening address using the RTMP protocol for pushing and pulling the audio and video streams.

[0200] S808, the conference signaling server synchronizes the monitoring address to the communication media server, and the communication media server notifies the signaling server that the media address and monitoring address reserved for the participant have been completed, so that the call intermediate number is connected.

[0201] S809, after the intermediate number is connected, the audio and video generated by the participants flow into the communication media server through the core network according to the media address, and the communication media server pushes the audio and video streams of the audio and video generated by the participants to the conference media server through the RTMP protocol.

[0202] S810: The conference media server mixes the audio and video streams corresponding to the audio and video to obtain a single-channel audio and video stream.

[0203] S811, according to the monitoring address, the communication media server pulls the single-channel audio and video stream from the conference media server and sends it to the participant, so as to conduct an audio and video conference based on the single-channel audio and video stream.

[0204] S812, when the participant exits the audio and video conference, the communication media server stops pushing and pulling the audio and video of the participant, and sends a notification of the participant exiting the conference to the conference signaling server, and removes the participant from the audio and video conference through the conference media server.

[0205] S813, when the audio and video conference ends, a conference end notification is sent to the communication media server through the conference signaling server and synchronized to the signaling server, and the conference end notification is sent to the participants through the signaling server.

[0206] Among them, the communication media server will close the audio and video conference and stop pushing and pulling streams for audio and video.

[0207] The specific process from S801 to S813 has been described in detail in the above embodiment and will not be repeated here.

[0208] By adopting the technical solution of the embodiment of the present application, the incoming signaling of the participant of the audio and video conference calling the middle number is received through the signaling server and sent to the communication media server, the communication connection with the conference signaling server is established through the communication media server, and the communication information is sent to the conference media server by using the conference signaling server, and the communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, the conference media server reserves a listening address for the audio and video and synchronizes it to the communication media server, the listening address is used to pull and push the audio and video streams of the audio and video, and when the middle number is connected, the conference media server mixes the audio and video to obtain a single-channel audio and video stream, and according to the listening address, the communication media server pulls the single-channel audio and video stream and sends it to the participant, so as to conduct an audio and video conference based on the single-channel audio and video stream. It can be seen that the participants of the meeting can connect to the audio and video conference by dialing the middle number. By reserving the media address and monitoring address for the audio and video corresponding to the participants, the audio and video streams corresponding to the audio and video can be pushed and pulled, allowing the participants to join the audio and video conference. The participants can enter the audio and video conference without downloading the audio and video conference software, thus solving the problem of low convenience in participating in audio and video conferences.

[0209] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0210] The above is a method for audio and video conferencing provided in an embodiment of the present application. Based on the same idea, an embodiment of the present application also provides a device for audio and video conferencing.

[0211] Fig. 9 FIG. 1 is a schematic diagram of the structure of an audio and video conferencing device according to an embodiment of the present invention. Fig. 9 As shown, the apparatus for audio and video conferencing includes: a receiving module 91, a creating module 92, a monitoring module 93, and a processing module 94:

[0212] The receiving module 91 is used to receive the incoming signaling of the participant of the audio and video conference calling the intermediate number through the signaling server, and send it to the communication media server;

[0213] A creation module 92, configured to establish a communication connection with a conference signaling server through a communication media server, and send communication information to the conference media server using the conference signaling server, the communication information including: a message of a participant joining the conference, and a media address reserved by the communication media server for audio and video of the incoming signaling;

[0214] The monitoring module 93 is used to reserve a monitoring address for audio and video through the conference media server according to the communication information, and synchronize it to the communication media server. The monitoring address is used to push and pull the audio and video streams of the audio and video;

[0215] Processing module 94 is used to mix the audio and video through the conference media server to obtain a single-channel audio and video stream when the intermediate number is connected. According to the monitoring address, the single-channel audio and video stream is pulled through the communication media server and sent to the participants to conduct an audio and video conference based on the single-channel audio and video stream.

[0216] In one embodiment, the apparatus further comprises:

[0217] Notification module 95, used to notify the communication media server to reserve a media address for the audio and video of the incoming signaling;

[0218] The determination module 96 is used to return the media address to the signaling server, and determine the media format of the participant's audio and video through the signaling server.

[0219] In one embodiment, the determination module 96 is specifically used to negotiate the media format of audio and video and the video through the signaling server and the participants when the incoming signaling is a voice call, and change the media format of audio and video to the negotiated media format.

[0220] In one embodiment, the apparatus further comprises:

[0221] A synchronization module 97, used to synchronize the audio and video information after determining the media format of the audio and video to the communication media server, and store it in the media address of the communication media server;

[0222] The first sending module 98 is used to send a start message for connecting the audio and video conference to the participants through the signaling server, and send the start message to the communication media server.

[0223] In one embodiment, the monitoring module 93 is specifically used for:

[0224] According to the communication information sent by the conference signaling server, the RTMP streaming media server including the RTMP protocol is started through the conference media server;

[0225] Based on RTMP streaming media server, a monitoring address is reserved for audio and video;

[0226] The monitoring address is returned to the conference signaling server through the conference media server, and the monitoring address is synchronized to the communication media server through the conference signaling server.

[0227] In one embodiment, the apparatus further comprises:

[0228] A second sending module 99 is used to send a reservation message to the signaling server using the communication media server, the reservation message including a message that the media address and the monitoring address are successfully reserved for the audio and video;

[0229] The third sending module 100 is used to send the reserved message to the participant through the signaling server.

[0230] In one embodiment, the processing module 94 is specifically used to receive audio and video through the conference media server, and perform mixing processing on the audio and video streams corresponding to the audio and video, and the mixing processing includes processing the audio and video streams corresponding to multiple audio and video channels into a single audio and video stream.

[0231] In one embodiment, the apparatus further comprises:

[0232] The acquisition module 101 is used to acquire the audio and video conference established by the user through the conference service front end, and send a request to create the audio and video conference to the conference signaling server;

[0233] A creation module 102, configured to send a request to a conference media server via a conference signaling server, and create a conference room using the conference media server;

[0234] The allocation module 103 is used to allocate an intermediate number to the conference room using the conference signaling server, and send the intermediate number to the conference service front end for display.

[0235] In one embodiment, the apparatus further comprises:

[0236] An exit module 104 is used for receiving an exit message of the participant exiting the audio and video conference through the signaling server when the participant exits the audio and video conference, and sending the exit message to the communication media server;

[0237] The first notification module 105 is used to stop the push and pull streaming of the participant's audio and video through the communication media server, and send a notification of the participant's exit from the conference to the conference signaling server;

[0238] The removal module 106 is used to send a notification of exiting the conference to the conference media server by using the conference signaling server, and remove the participant from the audio and video conference through the conference media server.

[0239] In one embodiment, the apparatus further comprises:

[0240] End module 107, used for sending a conference end notification to the communication media server through the conference signaling server when the audio and video conference ends, and notifying the communication media server to close the audio and video conference;

[0241] The second notification module 108 is used to stop the push and pull processing of the audio and video streams by using the communication media server, and synchronize the notification of the end of the conference to the signaling server;

[0242] The sending module 109 is used to send a notification of the end of the conference to the participants through the signaling server.

[0243] By adopting the technical solution of the embodiment of the present application, the incoming signaling of the participant of the audio and video conference calling the middle number is received through the signaling server and sent to the communication media server, the communication connection with the conference signaling server is established through the communication media server, and the communication information is sent to the conference media server by using the conference signaling server, and the communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, the conference media server reserves a listening address for the audio and video and synchronizes it to the communication media server, the listening address is used to pull and push the audio and video streams of the audio and video, and when the middle number is connected, the conference media server mixes the audio and video to obtain a single-channel audio and video stream, and according to the listening address, the communication media server pulls the single-channel audio and video stream and sends it to the participant, so as to conduct an audio and video conference based on the single-channel audio and video stream. It can be seen that the participants of the meeting can connect to the audio and video conference by dialing the middle number. By reserving the media address and monitoring address for the audio and video corresponding to the participants, the audio and video streams corresponding to the audio and video can be pushed and pulled, allowing the participants to join the audio and video conference. The participants can enter the audio and video conference without downloading the audio and video conference software, thus solving the problem of low convenience in participating in audio and video conferences.

[0244] Those skilled in the art should understand that Fig. 9 The audio and video conferencing device in the present invention can be used to implement the audio and video conferencing method described above, and the detailed description thereof should be similar to that described in the method section above. To avoid redundancy, it will not be further described here.

[0245] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to execute the above-mentioned audio and video conferencing method. Fig.10A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 1010, a communication interface (Communications Interface) 1020, a memory (memory) 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020, and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call a computer program stored in the memory 1030 and executable on the processor 1010 to perform the following steps:

[0246] Receive incoming signaling from a participant of an audio or video conference calling a middle number through a signaling server, and send it to a communication media server;

[0247] Establishing a communication connection with a conference signaling server through a communication media server, and using the conference signaling server to send communication information to the conference media server, the communication information including: a message of a participant joining the conference, and a media address reserved by the communication media server for audio and video of the incoming signaling;

[0248] According to the communication information, a listening address is reserved for audio and video through the conference media server and synchronized to the communication media server. The listening address is used to push and pull the audio and video streams of the audio and video;

[0249] When the middle number is connected, the audio and video are mixed through the conference media server to obtain a single-channel audio and video stream. According to the monitoring address, the single-channel audio and video stream is pulled through the communication media server and sent to the participants to conduct an audio and video conference based on the single-channel audio and video stream.

[0250] By adopting the technical solution of the embodiment of the present application, the incoming signaling of the participant of the audio and video conference calling the middle number is received through the signaling server and sent to the communication media server, the communication connection with the conference signaling server is established through the communication media server, and the communication information is sent to the conference media server by using the conference signaling server, and the communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, the conference media server reserves a listening address for the audio and video and synchronizes it to the communication media server, the listening address is used to pull and push the audio and video streams of the audio and video, and when the middle number is connected, the conference media server mixes the audio and video to obtain a single-channel audio and video stream, and according to the listening address, the communication media server pulls the single-channel audio and video stream and sends it to the participant, so as to conduct an audio and video conference based on the single-channel audio and video stream. It can be seen that the participants of the meeting can connect to the audio and video conference by dialing the middle number. By reserving the media address and monitoring address for the audio and video corresponding to the participants, the audio and video streams corresponding to the audio and video can be pushed and pulled, allowing the participants to join the audio and video conference. The participants can enter the audio and video conference without downloading the audio and video conference software, thus solving the problem of low convenience in participating in audio and video conferences.

[0251] The specific execution steps can refer to the various steps of the above-mentioned audio and video conferencing method implementation example, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0252] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.

[0253] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0254] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0255] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0256] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned audio and video conferencing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0257] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0258] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio and video conferencing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0259] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0260] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the processor is used to run the program or instructions to implement the various processes of the above-mentioned product recommendation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0261] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0262] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0263] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for audio and video conferencing, characterized in that: The method comprises: Receive incoming signaling from a participant of an audio or video conference calling a middle number through a signaling server, and send it to a communication media server; Creating a communication connection with a conference signaling server through the communication media server, and using the conference signaling server to send communication information to the conference media server, the communication information including: a message of the participant joining the conference, and a media address reserved by the communication media server for the audio and video of the incoming signaling; According to the communication information, a monitoring address is reserved for the audio and video through the conference media server and synchronized to the communication media server, and the monitoring address is used to push and pull the audio and video streams of the audio and video; When the intermediate number is connected, the audio and video are mixed through the conference media server to obtain a single-channel audio and video stream. According to the monitoring address, the single-channel audio and video stream is pulled by the communication media server and sent to the participant to conduct the audio and video conference based on the single-channel audio and video stream.

2. The method according to claim 1, characterized in that After receiving the incoming signaling of the participant of the audio and video conference calling the intermediate number through the signaling server and sending it to the communication media server, it also includes: Notify the communication media server to reserve the media address for the audio and video of the incoming signaling; The media address is returned to the signaling server, and the media format of the audio and video of the participant is determined through the signaling server.

3. The method according to claim 2, characterized in that The determining, by the signaling server, the media format of the audio and video of the participant includes: In the case that the incoming signaling is a voice incoming call, the media format of the audio and video is negotiated and the video is negotiated by the signaling server and the participant, and the media format of the audio and video is changed to the negotiated media format.

4. The method according to claim 2, characterized in that: Before the communication media server creates a communication connection with the conference signaling server, the method includes: Synchronize the audio and video information after determining the media format of the audio and video to the communication media server, and store it in the media address of the communication media server; A start message for connecting the audio and video conference is sent to the participant through the signaling server, and the start message is sent to the communication media server.

5. The method according to claim 1, characterized in that The method of reserving a monitoring address for the audio and video through the conference media server according to the communication information and synchronizing the address to the communication media server includes: According to the communication information sent by the conference signaling server, starting an RTMP streaming media server including a real-time messaging protocol RTMP protocol through the conference media server; Based on the RTMP streaming media server, reserving the monitoring address for the audio and video; The monitoring address is returned to the conference signaling server through the conference media server, and the monitoring address is synchronized to the communication media server through the conference signaling server.

6. The method according to claim 1, characterized in that After reserving a monitoring address for the audio and video by the conference media server, the method includes: Using the communication media server to send a reservation message to the signaling server, the reservation message includes a message that the media address and the monitoring address are successfully reserved for the audio and video; According to the reservation message, it is sent to the participant through the signaling server.

7. The method according to claim 1, characterized in that The mixing of the audio and video by the conference media server to obtain a single-channel audio and video stream includes: The audio and video are received by the conference media server, and the audio and video streams corresponding to the audio and video are mixed, wherein the mixing includes processing the audio and video streams corresponding to the multiple channels of the audio and video into the single channel of the audio and video stream.

8. The method according to claim 1, characterized in that Before the signaling server receives an incoming signaling from a participant of an audio or video conference calling an intermediate number, the method includes: Acquire the audio and video conference established by the user through the conference service front end, and send a request to the conference signaling server to create the audio and video conference; Sending the request to the conference media server through the conference signaling server, and creating a conference room using the conference media server; The conference signaling server is used to allocate the intermediate number to the conference room, and the intermediate number is sent to the conference service front end for display.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: When the participant exits the audio and video conference, receiving an exit message of the participant exiting the audio and video conference through the signaling server, and sending the exit message to the communication media server; Stopping the push and pull streaming of the audio and video of the participant through the communication media server, and sending a notification of the participant exiting the conference to the conference signaling server; The conference signaling server is used to send the notification of exiting the conference to the conference media server, and the participant is removed from the audio and video conference through the conference media server.

10. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: When the audio and video conference ends, sending a conference end notification to the communication media server through the conference signaling server, and notifying the communication media server to close the audio and video conference; Using the communication media server to stop the push and pull processing of the audio and video, and synchronizing the notification of the end of the conference to the signaling server; A notification of the end of the conference is sent to the participants through the signaling server.

Images